Info Favorites Register Log in
myArmoury.com Discussion Forums

Forum index Memberlist Usergroups Spotlight Topics Search


Please help our efforts with a donation. This site requires ongoing funding and your donations are crucial to our future.
Last 10 Donors: Anonymous, Daniel Sullivan, Chad Arnow, Mateusz S., J.D. Crawford, Eric S, Håvard Kongsrud, Jakub Malovany, M. Livermore, Tim Mathews (View All Donors)

Forum Index > myArmoury.com Features Talk > FIXED: Photobucket image hosting Reply to topic
This is a standard topic Go to page 1, 2  Next 
Author Message
Nathan Robinson
myArmoury Admin


myArmoury Admin

PostPosted: Thu 20 Jul, 2017 11:49 pm    Post subject: FIXED: Photobucket image hosting         Reply with quote

What is Photobucket and why does myArmoury.com need a fix?

Many people wishing to share images on the 'net use the popular image hosting site Photobucket to host images that can then be posted in topics on forums like myArmoury.com. They provided this service free of charge, and for a little bit of cash would allow their users a bunch of bandwidth as well. Recently they changed their business model and now want to charge members $400/annually to allow for their photos to be linked and shown on other web sites.

This decision breaks millions of forum topics all over the Internet, eBay auctions, and other pages that have had images from Photobucket on them, showing this image instead of the one intended:



You can read more about the problem here

This was particularly upsetting to me given that I own this site and suddenly in one fell swoop hundreds of topics here were suddenly shown with broken images! We've had some great things posted here so I had to do something.


Can't something be done to fix this?

With a bit of programming magic I came up with a solution that now correctly displays the Photobucket hosted images in our topics!

Check out this topic here as an example

That topic contains a lot of images hosted on Photobucket that would show up as broken until my newly implemented fix.


So what?

Topics are hard to read when they're full of broken images. A lot of our really great discussions here are image-intensive and really rely on those images to tell the whole story. Without them, a significant percentage of our content's value is diminished.

I think myArmoury.com might be the only forum to have even tried to tackle this problem at this time. I believe I'd bet I'm the first to come up with a solution. I won't reveal how I made this fix but will say that my solution is pretty creative if I say so myself.

This fix is still being tested so there are bound to be some issues with it.


Can I see some examples?

All the photos below are hosted on Photobucket. Without the fix, they would all show up as broken images. They're displaying correctly now (assuming that the people don't delete them from their Photobucket accounts, of course!)
















.:. Visit my Collection Gallery :: View my Reading List :: View my Wish List :: See Pages I Like :: Find me on Facebook .:.
View user's profile Send private message Send e-mail Visit poster's website
Greyson Brown




Location: Windsor, Colorado
Joined: 22 Nov 2004
Reading list: 15 books

Posts: 790

PostPosted: Fri 21 Jul, 2017 7:46 am    Post subject:         Reply with quote

$400 annually !?! That is highway robbery. You could pay for actual site hosting and upload pictures for the purpose of linking so much cheaper (maybe even free depending on the WordPress.com terms of use).

You said you won't share how you worked around this problem so I won't actually ask that, but I will say that it obviously took some research and hard work. Adding a database trigger to update the links is the obvious part, but figuring out what the link needs to be is still eluding me. I don't really have a need to know how this was done, but I may waste part of the day trying to figure it out anyway.

Thanks for taking the time to figure this out. You are right that many topics on this site lose a lot of value without working images.

-- Greyson

"So long as I can keep the path of honor I am well content."
-Sir Arthur Conan Doyle, The White Company
View user's profile Send private message
Stephen Curtin




Location: Cork, Ireland
Joined: 17 Nov 2007
Likes: 110 pages
Reading list: 18 books

Spotlight topics: 1
Posts: 1,136

PostPosted: Fri 21 Jul, 2017 8:28 am    Post subject:         Reply with quote

As always Nathan your efforts are much appreciated.
Éirinn go Brách
View user's profile Send private message Send e-mail
Mikko Kuusirati




Location: Finland
Joined: 16 Nov 2004
Reading list: 13 books

Posts: 931

PostPosted: Fri 21 Jul, 2017 11:24 am    Post subject:         Reply with quote

Applause, sir! Uproarious applause!

I moved to Imgur as soon as Photobucket started spamming me with honest-to-gods extortion emails some weeks ago ("Nice photos you've shared, be a real shame if anything happened to them," and embarrassingly close to those actual words) but of course that doesn't help all the old threads on all the forums I've posted images in and can't edit anymore...

I'm fairly impressed and quite curious how you did that. But if you don't wanna spill, eh, fair I guess. Happy

The subtle tongue, the sophist guile, they fail when the broadswords sing;
Rush in and die, dogs -- I was a man before I was a king.
-- R. E. Howard, The Road of Kings
View user's profile Send private message
Daniel Sullivan




Location: California
Joined: 02 Apr 2004
Likes: 9 pages
Reading list: 3 books

Posts: 200

PostPosted: Fri 21 Jul, 2017 12:58 pm    Post subject: FIXED Photobucket image hosting         Reply with quote

Nathan,

Wonderful! Frankly, I was getting a bit depressed over this situation as I believed the site had suffered a serious wound. Now I believe in "miracles" and suspect there might even be a "higher authority" .....

A well earned Thank You!

Regards.
Dan
View user's profile Send private message
Luka Borscak




Location: Croatia
Joined: 11 Jun 2007
Likes: 7 pages

Posts: 2,216

PostPosted: Fri 21 Jul, 2017 1:05 pm    Post subject:         Reply with quote

I checked my latest review and all pictures are there it seems. I don't know how you did it, but thank you. Happy
View user's profile Send private message
Nathan Robinson
myArmoury Admin


myArmoury Admin

PostPosted: Fri 21 Jul, 2017 1:09 pm    Post subject:         Reply with quote

Luka Borscak wrote:
I checked my latest review and all pictures are there it seems. I don't know how you did it, but thank you. Happy


Occasionally, when I'm not being foolish, I can be pretty clever.

Your recently posted-review was one of the main reasons I put the time in to figure out a hack to fix this problem here.

.:. Visit my Collection Gallery :: View my Reading List :: View my Wish List :: See Pages I Like :: Find me on Facebook .:.
View user's profile Send private message Send e-mail Visit poster's website
Greyson Brown




Location: Windsor, Colorado
Joined: 22 Nov 2004
Reading list: 15 books

Posts: 790

PostPosted: Fri 21 Jul, 2017 3:24 pm    Post subject:         Reply with quote

Nathan,

My hat is off to you for finding a work around for this without throwing computers out the window. I have spent a total of maybe 3 hours looking into this today and I can feel my blood pressure rising. I created a junk photobucket account and uploaded a picture. I could see my picture and link to it from an external site for about 20 minutes; after that I couldn't even view my own picture when logged into photobucket. From within their own site, I got the "update your account to allow 3rd party hosting" message! They seem to not understand the concept of "3rd party" (or the difference between update and upgrade).

You, sir, have my respect for being able to tolerate their festering, ad-filled excuse for a site long enough to solve this problem.

-- Greyson

"So long as I can keep the path of honor I am well content."
-Sir Arthur Conan Doyle, The White Company
View user's profile Send private message
Aaron Hoard




Location: Seattle, WA
Joined: 01 Sep 2009
Likes: 4 pages

Posts: 147

PostPosted: Fri 21 Jul, 2017 5:13 pm    Post subject:         Reply with quote

Thank you for working to resolve this.
View user's profile Send private message
Lin Robinson




Location: NC
Joined: 15 Jun 2006
Likes: 6 pages
Reading list: 6 books

Posts: 1,184

PostPosted: Sat 22 Jul, 2017 6:30 am    Post subject:         Reply with quote

Nathan Robinson wrote:
Luka Borscak wrote:
I checked my latest review and all pictures are there it seems. I don't know how you did it, but thank you. Happy


Occasionally, when I'm not being foolish, I can be pretty clever.

Your recently posted-review was one of the main reasons I put the time in to figure out a hack to fix this problem here.


I will echo what the others said and extend my thanks to you for the effort expended and sympathize about the angst you undoubtedly suffered when this happened. I have used Photobucket from time to time on this and other forums but I don't think I will any more. I just went to my account to see if my photos were still there and they are, but I have them elsewhere as well. What I did notice were a large number of ads inserted in my catalog of photos and an offer to eliminate the ads for $2.49 per month. Not taking that offer either. IMHO many of these companies, once they get their infrastructure and staff in place, don't have a lot of other expense and ploys like this are simply to increase their profits on the backs of folks that they think must use their services. Wrong about that, at least in my case. I can live without them.

Lin Robinson

"The best thing in life is to crush your enemies, see them driven before you and hear the lamentation of their women." Conan the Barbarian, 1982
View user's profile Send private message
Nathan Robinson
myArmoury Admin


myArmoury Admin

PostPosted: Sat 22 Jul, 2017 4:36 pm    Post subject:         Reply with quote

Hi Everybody-

I did some statistical analysis of the Photobucket images that have been posted on our forum:

Number of PB images posted: 62,233
Distinct number of PB images posted: 56,481


Since the fix we have already served up 12,228 distinct images at this time. These images have been cached locally in case PB changes their technology, moves the images, or goes out of business. This number will just keep growing as posts are accessed.

.:. Visit my Collection Gallery :: View my Reading List :: View my Wish List :: See Pages I Like :: Find me on Facebook .:.
View user's profile Send private message Send e-mail Visit poster's website
Jonathan Hopkins




PostPosted: Sat 22 Jul, 2017 6:56 pm    Post subject:         Reply with quote

Well done, and thank you, Nathan!
View user's profile Send private message
Nathan Robinson
myArmoury Admin


myArmoury Admin

PostPosted: Thu 27 Jul, 2017 8:21 pm    Post subject:         Reply with quote

Since the fix we have saved 21,549 distinct images from Photobucket's prison.

These images have been cached locally in case PB changes their technology, moves the images, or goes out of business. This number will just keep growing as posts are accessed.

.:. Visit my Collection Gallery :: View my Reading List :: View my Wish List :: See Pages I Like :: Find me on Facebook .:.
View user's profile Send private message Send e-mail Visit poster's website
Nathan Robinson
myArmoury Admin


myArmoury Admin

PostPosted: Thu 17 Aug, 2017 11:12 am    Post subject:         Reply with quote

Since the fix we have saved 39,431 distinct images from Photobucket's prison.

These images have been cached locally in case PB changes their technology, moves the images, or goes out of business. This number will just keep growing as posts are accessed.

This is a whole lot of content we saved from the ashes.

.:. Visit my Collection Gallery :: View my Reading List :: View my Wish List :: See Pages I Like :: Find me on Facebook .:.
View user's profile Send private message Send e-mail Visit poster's website
Greyson Brown




Location: Windsor, Colorado
Joined: 22 Nov 2004
Reading list: 15 books

Posts: 790

PostPosted: Fri 18 Aug, 2017 6:36 am    Post subject:         Reply with quote

Nathan Robinson wrote:
Since the fix we have saved 39,431 distinct images from Photobucket's prison.

These images have been cached locally in case PB changes their technology, moves the images, or goes out of business. This number will just keep growing as posts are accessed.

This is a whole lot of content we saved from the ashes.


Perhaps a pointless suggestion, but have you considered querying the database to pull all those post that have photobucket images and then putting those URLs in a script to open those pages? That way you could ensure that all of the images are backed up without waiting for people to manually access those posts.

-- Greyson

"So long as I can keep the path of honor I am well content."
-Sir Arthur Conan Doyle, The White Company
View user's profile Send private message
Julien M




Location: London
Joined: 14 Sep 2005

Spotlight topics: 1
Posts: 1,053

PostPosted: Fri 18 Aug, 2017 9:53 am    Post subject:         Reply with quote

I noticed the pics were live again and naively thought Photobucket had second thoughts about their rather unfriendly conduct...but could not find anything to corroborate that hypothesis online - I can see why now Happy
Well played Nathan - would have been a shame to loose all that content!
I'll stick with Imgur from now on...
J
View user's profile Send private message
Nathan Robinson
myArmoury Admin


myArmoury Admin

PostPosted: Sun 20 Aug, 2017 7:00 pm    Post subject:         Reply with quote

Greyson Brown wrote:
have you considered querying the database to pull all those post that have photobucket images and then putting those URLs in a script to open those pages? That way you could ensure that all of the images are backed up without waiting for people to manually access those posts.



I have considered it but could not think of a good way to make this happen. Instead, I relied on the fact that web crawlers by Google, Yahoo, etc, index our site daily. I can log into my Google search console and request a reindex with a supplied sitemap. I created a sitemap with all the topics and requested the reindex. This essentially does the same thing as you're proposing.

.:. Visit my Collection Gallery :: View my Reading List :: View my Wish List :: See Pages I Like :: Find me on Facebook .:.
View user's profile Send private message Send e-mail Visit poster's website
Greyson Brown




Location: Windsor, Colorado
Joined: 22 Nov 2004
Reading list: 15 books

Posts: 790

PostPosted: Mon 21 Aug, 2017 7:21 am    Post subject:         Reply with quote

Nathan Robinson wrote:
Greyson Brown wrote:
have you considered querying the database to pull all those post that have photobucket images and then putting those URLs in a script to open those pages? That way you could ensure that all of the images are backed up without waiting for people to manually access those posts.



I have considered it but could not think of a good way to make this happen. Instead, I relied on the fact that web crawlers by Google, Yahoo, etc, index our site daily. I can log into my Google search console and request a reindex with a supplied sitemap. I created a sitemap with all the topics and requested the reindex. This essentially does the same thing as you're proposing.


I would do it by using a query that builds the commands needed for a script. I use Linux almost exclusively, so I would use:

SELECT concat("firefox http://myArmoury.com/talk/viewtopic.php?t=",topicid," &")
FROM posts_table
WHERE post_content LIKE '%photobucket%';

(replace firefox with browser of choice, and, of course, provide correct column and table names).

I would then create a script (just a text file with permission to execute it) that contains the output from the above query (just copy and paste) followed by:
sleep 1m # wait a little bit just to ensure the pages have time to load
pkill -f firefox # close firefox when done

Run the script while you eat lunch or something where the computer is not actively in use as all of those browser windows opening would get annoying if you are trying to work.

It has been a while since I have done Windows batch files, but I think the only changes are that the & at the end of each line is not needed, sleep is not supported (the work around is to ping 127.0.0.1 60 times (each ping is close to 1 second)), and pkill is replaced with taskkill.

Your solution works just fine, though.

"So long as I can keep the path of honor I am well content."
-Sir Arthur Conan Doyle, The White Company
View user's profile Send private message
Greyson Brown




Location: Windsor, Colorado
Joined: 22 Nov 2004
Reading list: 15 books

Posts: 790

PostPosted: Mon 21 Aug, 2017 7:29 am    Post subject:         Reply with quote

BTW: My previous post assumed a MySQL database, as that is the most likely. For Microsoft SQL Server, change
SELECT concat("firefox http://myArmoury.com/talk/viewtopic.php?t=",topicid," &")
to SELECT "firefox http://myArmoury.com/talk/viewtopic.php?t="+topicid+" &"

For Oracle, it would be:
SELECT "firefox http://myArmoury.com/talk/viewtopic.php?t="||topicid||" &"

"So long as I can keep the path of honor I am well content."
-Sir Arthur Conan Doyle, The White Company
View user's profile Send private message
Nathan Robinson
myArmoury Admin


myArmoury Admin

PostPosted: Thu 24 Aug, 2017 4:32 pm    Post subject:         Reply with quote

Hey Greyson, I don't want to clutter this topic with technical stuff other than to say this: I know how to make a script that queries the pages, but there's a lot of other complications such as topic pagination (digging down into multi-page topics), passing session variables/cookies, server load, and some other database-specific stuff I don't want to mention here. At this point, it's moot because...

~~~~~~~~

Since the fix we have saved 51,228 distinct images from Photobucket's prison.

These images have been cached locally in case PB changes their technology, moves the images, or goes out of business. This number will just keep growing as posts are accessed.

We've saved 90% of the 56,481 hosted images.

.:. Visit my Collection Gallery :: View my Reading List :: View my Wish List :: See Pages I Like :: Find me on Facebook .:.
View user's profile Send private message Send e-mail Visit poster's website


Display posts from previous:   
Forum Index > myArmoury.com Features Talk > FIXED: Photobucket image hosting
Page 1 of 2 Reply to topic
Go to page 1, 2  Next All times are GMT - 8 Hours

View previous topic :: View next topic
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum






All contents © Copyright 2003-2018 myArmoury.com — All rights reserved
Discussion forums powered by phpBB © The phpBB Group
Switch to the Basic Low-bandwidth Version of the forum