Sciencemadness Discussion Board

Site Backup

Panache - 9-4-2009 at 17:56

Is the site still being backed-up and being made available?

EDIT BY POLVERONE JUNE 2012: Please skip to the end of the thread for latest instructions. The forum backup is now being distributed as a virtual machine appliance.

[Edited on 2-2-2020 by Polverone]

Panache - 11-4-2009 at 17:58

Archive.org stopped backing up this forum Feb 2008, over 12 months ago, does anyone know of anything archiving site that does this for this site?

Polverone - 12-4-2009 at 11:54

I still download site backups pretty frequently. The last one was about a month ago. I have unfortunately not updated the downloadable site archive in quite some time. The problem is that the downloadable version needs many small HTML changes to fix links and page organization. Each time the XMB version changes I need to change the programs that do these alterations. It is a tedious and slow process to update the programs.

It probably wouldn't be that hard to distribute a preconfigured Linux disk image for a virtual machine that has XMB already installed and just needs a fresh database dump to make it match Sciencemadness. If someone wants to make such an image I could provide sanitized database dumps so you can run your own local backup instance of sciencemadness. The sanitized dump would be missing U2U messages, member email addresses and passwords, and the contents of Whimsy and References but otherwise would match the live forum.

I could make such a disk image myself but it would take time that I am short of, which is the same reason I have not updated the conventional forum backup in so long. If there are any tech-savvy members who will take the initiative to produce the disk image, I promise that I will meet them half way and provide the sanitized database dumps.

Any Linux image for this purpose should have recent versions of Apache, MySQL, and PHP installed. It should also have the latest (1.9.11) version of XMB, Firefox or Opera, a PDF reader, a djvu reader, and host/guest integration tools for the selected virtualization platform (VMWare Player or (preferred) VirtualBox). Otherwise the image should probably be slimmed down as much as possible.

hodges - 6-9-2009 at 15:22

Not sure if this is still needed, but I have a Ubuntu Linux image for VMWare with MySQL and PHP5 installed. So I was playing around with it today and installed XMB 1.9.11 and set up a database according to the instructions in the install. I was able to get a new bulletin board set up (with no content of course, just the default screens) fairly quickly.

If you wish, I can upload this image. Or, if you want to give me sanitized data to load I can load it first then upload.

The image can be played with VMWare Player, which is free for both Windows and Linux. The Ubuntu version has all the latest patches, and is supported until 2011 (at which time the version probably needs to be upgraded). It includes Firefox, as well as a viewer for PDF and DJVU files. The size of the zip file, including the VMWare Player for windows, is around 1GB, with the unzipped size being about 2.5GB. The data disk is configured to grow up to 8GB as required based on the data stored.

Let me know if you are interested.

Hodges

MagicJigPipe - 29-1-2010 at 10:11

Where can I find the most recent backup? The search engine only reveals one from 2008.

Thanks.

[Edited on 1-29-2010 by MagicJigPipe]

Polverone - 30-1-2010 at 12:21

Hodges, I am sorry that I missed your post earlier. I did almost exactly what you did, except I started out with a compact Turnkey Linux image and used VirtualBox instead of VMWare. I now have a distributable Linux-image that contains a live, runnable (but sanitized) version of the forum. Details following in next post.

Live, runnable backup copy of Sciencemadness forum now available

Polverone - 30-1-2010 at 12:25

If you want a backup copy of the SM forum that can be run on your own computer, start downloading this file:

Offline Copy of Sciencemadness

It is big, about 1.8 GB. A successful download should have the MD5 hash 6a41b7a28c999a243a388f6db5d83d62. Use a tool that automatically resumes interrupted downloads, like wget or GetRight, to download it.

After you have downloaded the archive itself, you will also want to download and install VirtualBox, the free virtualization software. The .ovf format supposedly works with other virtualization software too, like VMWare, but I have not tried to validate this.

Unzip the sm_offline.zip, then import the .ovf file from the contents with VirtualBox's Import Appliance wizard:




The import process may take several minutes to complete.



After you have finished importing the image, be sure to set the "Enable IO APIC" feature under the virtual machine's system settings. If this feature is not enabled, the virtual machine will fail to start properly.



After IO APIC has been enabled, you should be able to press "start" for the virtual machine and see this virtual console appear after a few seconds of startup activity:



You will want to point your web browser to the same IP address as shown in the console. You should see a screen like this, being served from the virtual machine's web server:



If you click on the most prominent link, you should see the forum, looking much as it did at the time the backup copy was made:



You can browse down into all the saved forums:



When you are done, go back to the virtual console and choose "Shutdown" from the advanced menu:



Privacy

I have removed all U2U messages, posts from protected forums, and attachments that belonged to the removed posts. Members whose email addresses are not set to display have had their addresses blanked out. Invisible members have been made visible. All IP address information has been removed. All member passwords have been changed to 'reader'. Buddy, favorite, and ignore lists have been removed.

Passwords
As mentioned above, the password for every forum account in the backup is 'reader.' The root password on the virtual machine is 'science'.

Usage
This backup can be browsed and searched without a working internet connection. Should Sciencemadness ever disappear, this backup should retain all of the public forum posts. What's more, the actual XMB software configuration and MySQL database contained in the virtual image could be used to restart the forum on another web host if I became unable to maintain this site.

It is even possible, though probably pointless, to log in to the backup as an existing member and then post messages to the backup itself. If you are curious about the administrator's powers, you can even log in as Polverone with the same password. This gives you, for example, access to the admin control panel where you can search for attachments (a feature not available to ordinary members on the live site).

[Edited on 2-2-2010 by Polverone]

dann2 - 24-2-2010 at 06:05

Hello,
On attempting to download the 1.72Gb file with Wget, I managed to get to 46% before the thing 'give up'. (It's own description). There were many 'timeouts', and stops and starts along the way to the 46% mark and it resumed from where it started OK.
When I cranked up Wget again it starts to download to a new file from the beginning. Is there any way around this?

EDIT:
Good afternoon Dann2!
Use the -c switch (you silly buffoon).
eg: (at command prompt)...> wget -c http://library.sciencemadness.org/archive/sm_offline.zip

Thanks,
Dann2

To admin.
Delete this post of you so wish. I left it here is it may be useful.

[Edited on 24-2-2010 by dann2]

dann2 - 25-2-2010 at 15:02


Hello,

Is anyone having problems with the Sun Virtual Machine.
I cannot get it to run.
It somes up with a 'No Interfaces configured' message box.
During the start up (after I press the green start arrow) it says (amongst other things) that the lease has failed.
Does this ring any bells?
I have the sm-offling.ovf file and the virtual box installed on an external hard drive if that makes any difference.

Dann2

photos.jpg - 70kB

[Edited on 25-2-2010 by dann2]

error.jpg - 5kB

Polverone - 26-2-2010 at 16:39

Hi dann2,

I have been able to reproduce your problem. At home my computers are connected to a router. If I connect a computer to the internet directly, no router, or unplug the network connection altogether, I get the same error you saw when I try to start the VM. I do not know a lot about networking but I will try to find out how this problem can be solved.

watson.fawkes - 26-2-2010 at 17:40

Quote: Originally posted by Polverone  
If I connect a computer to the internet directly, no router, or unplug the network connection altogether, I get the same error you saw when I try to start the VM.
The DHCP client is failing, meaning it's trying to get an IP address from a DHCP server and failing. Without an IP address, you can't get a server to bind to a port. In the present situation, you don't need any network address other than localhost, 127.0.0.1. The relevant interface could be manually configured that way with "ifconfig" at boot time, replacing the use of the DHCP client.

Polverone - 26-2-2010 at 18:06

Quote: Originally posted by watson.fawkes  
The DHCP client is failing, meaning it's trying to get an IP address from a DHCP server and failing. Without an IP address, you can't get a server to bind to a port. In the present situation, you don't need any network address other than localhost, 127.0.0.1. The relevant interface could be manually configured that way with "ifconfig" at boot time, replacing the use of the DHCP client.


I tried manually configuring it with 127.0.0.1, netmask 255.255.255.0. No success. I also tried with 127.0.1.1 since a boot message indicated that was being used as the host name -- still no luck.

The VM is set up with bridged networking because the network services (MySQL and HTTP) need to be accessible from the host machine, and all the VM guides I came across indicated that bridged networking was the way to go for that. I don't know why it works so easily with DHCP available but so dismally when I try to configure a static address. I will try a couple more things to get static working, and if that fails I'm going to see if running a DHCP server on the host machine can fix it.

dann2 - 26-2-2010 at 18:26

Hello,

Ran the Virtual Box again and got the 'No interfaces configured'.
Then went to configure the NIC manually (a box came up asking me).

I used 127.0.1.1 for three of the entries except gateway where I used 0.0.0.0 (wild assed guess)

That got me a bit further.............untill the yoke started to look for a lamp login and password.
I tried some rubbish logins and passwords to no avail.
I tried 'science' for login and 'reader' for password to no avail.


I decided to start again.
Things went different this time (for some unknown reason).
I did not get the ' No interfaces are configured' message but got the box as per the picture depicting the IP Address to use (two of them the same)
+ Webmin http address
+ PHPmyadmin
+ SSH
(similar stuff to picture way above only differetn actual numbers).

I put the http address into my browser address bar and get the next correct screen shown
Turnkey linix application running etc etc (same as picture way above )

I click on ScienceMadness Forum Archve but get a 'Internet Explorer Cannot display web page' (standard message from MS Explorer).


Thanks for your time.

Dann2

Nearly lost that post only for my habit of select and copy before hitting post or preview. :o:o:o:o:o:o:o:o:o

[Edited on 27-2-2010 by dann2]

Polverone - 26-2-2010 at 18:37

If you click on 'phpinfo' do you get some information displayed, or an error from the browser? If the phpinfo displays correctly I think I know why the forum is not working. The forum configuration file needs to be updated at startup-time with the address that you will use to get to the forum. I added a script that performs this update when the Turnkey Linux configuration screen first comes up. Since you configured the network interface after this interface program began running, the forum configuration file will not be properly updated.

If the phpinfo screen is not working, though, I am afraid the problem is deeper than forum configuration.

dann2 - 26-2-2010 at 18:48

Hello,

I tried again this time with my network cable plugged in. It was not plugged in the last time as I purposly plugged in out when running the Virtual box.

With the cable plugged in all runs OK and I can read all of forum and post to 'SciMadness forum' as Polverone!! and Dann2 as well (I suppose) though I did not try.

I aslo get large long page of info when PHPINFO is clicked.

Dann2

dann2 - 26-2-2010 at 18:52

Hello,

I attached the phpinfo page if you want it.

Why do I need to network cable plugged in the back (my interned connection). Sounds very iffey.

Dann2

Attachment: phpinfo().mht (70kB)
This file has been downloaded 2535 times

watson.fawkes - 27-2-2010 at 06:02

Quote: Originally posted by Polverone  
I tried manually configuring it with 127.0.0.1, netmask 255.255.255.0. No success. I also tried with 127.0.1.1 since a boot message indicated that was being used as the host name -- still no luck.
If it needs to route between these two addresses (and I don't know if it does or not), the netmask would need to be 255.255.0.0, since otherwise the mask would prevent the two local-network addresses (127.*) from being seen as the same network. (A manual route would work in that case, but the netmask is better.)

Running a DHCP server in the virtual box, though, sounds like it might be the quickest solution.

Rosco Bodine - 11-10-2010 at 14:51

What is the most recent backup? A lot of information has been added in the past few months, so a current backup would be timely.

Polverone - 12-10-2010 at 21:22

Thanks for the reminder. I have replaced the previous backup with one created yesterday. The link is still http://library.sciencemadness.org/archive/sm_offline.zip

Rosco Bodine - 15-10-2010 at 06:52

You are welcome. The attachments are a very important part of the database. Is there a way to include the attachments database so that attachments are available from the backup messages ?

Under her clothes this doll should be 100% anatomically correct :D :cool:

http://www.youtube.com/watch?v=Qrw5qEDRLw0&fmt=18

[Edited on 15-10-2010 by Rosco Bodine]

Polverone - 15-10-2010 at 12:14

The attachments in public forums are part of the backup. If files were posted to external file hosting sites instead of directly attached they of course are not contained in the backup, though the links are still there and can be downloaded if the file host retained the file.

Files that were uploaded to the scipics account are not part of the backup either, but that is easy enough to back up. Just use a tool like wget with its mirror option to get everything in http://www.sciencemadness.org/scipics/. I have taken no measures to hide that directory or limit downloads from it. If you'd prefer I can include a scipics copy as part of future backups, though that will add several hundred megabytes.

I will not release backups containing the contents of the restricted forums as long as this site remains online. In the event that the site were shut down, I could release a VM image that contains those forums too from one of my more frequent private backups. Or, if you want to get really thorough about information survival, I could distribute encrypted full backups and leave the password with other moderators to distribute in case SM goes offline and I am unable to respond. That way in case of crisis some tech-savvy moderators or members can reboot the forum with most data intact on another web host, or even running as a TOR onion service for extreme survivability.

I am not going to distribute backups (even encrypted ones) that contain user U2U messages and passwords under any circumstances, so when I speak of "full" backups I just mean ones with all posts and attachments intact.

EDIT: be careful with using mirroring software like wget that you do not allow it to move back up the directory hierarchy and try to get the whole forum. If you do allow it to run away, and you have a fast internet connection, and you haven't set your options to delay or throttle downloads, you may get locked out of the site for up to 24 hours by an automatic protection script I have in place. I managed to lock myself out a few minutes ago while validating the scipics backup method, until I directly logged into the host machine and reset the restrictions.

[Edited on 10-15-2010 by Polverone]

Rosco Bodine - 15-10-2010 at 16:26

Yeah, I think the scipics directory should be included as part of the forum backup.
There has been suggestion that files be put there to preserve them as part of the forum archive so that images and files don't get lost or expired by external hosting sites. Especially the member publications section would suffer from missing images content. The pdf conversion files for those threads generally exceed the direct attachment file size limit and have to be uploaded to the scipics directory anyway. If the search engine and printable version functions are possible to be included or are already in the backup that would be great also.

Polverone - 15-10-2010 at 18:16

The VM-image backups are actually running the forum software, so you can use the forum software's search tool (such as it is) and get printable versions of threads. I will include the scipics contents in the next backup.

sciencemadness contingency plan

Polverone - 24-12-2010 at 11:10

I have uploaded a new virtual machine image containing a sanitized snapshot of the forum database and all the supporting software needed to run it. This backup is much larger than the last one I made available because it incorporates the contents of the scipics directory plus encrypted copies of all posts and attachments from the private forums. Even in the protected private archives, though, I have removed personal information such as account password hashes, private email addresses, U2U messages, and IP addresses.

The new backup also contains scripts and instructions that should allow anyone with the archive password to use the VM image contents as a seed to restart the forum on a new web host, with a new domain name, if sciencemadness.org and/or I ever disappear. I am certainly not expecting to disappear, but I have seen other popular forums vanish for no good reason and I want to ensure that it can never happen to this site. This site is really made by the members who participate here, and I want to forestall the possibility of it ever being lost to them.

I don't know who here would be comfortable with the technical aspects of trying to revive the forum in the event of my disappearance. I know that some of our members must be technically capable. I want to spread the forum backup as far as possible, but I am leaving the password needed to recover the private forums in the hands of the moderators. They are asked to spread the password far and wide should I fall out of contact for more than 30 days.

As before, you should use a download client with resume-capability to get the latest offline copy of Sciencemadness. The instructions for bringing the virtual image to life are still the same. There is now a recovery guide included in the image, accessible via your web browser, explaining how to re-integrate the protected forum contents and reseed it on a new web host if you have the password.

You might want click on the printable version of this thread and print it to PDF if you haven't previously installed a VirtualBox image, since in an emergency this thread might not be available for reference.

I will try to keep the VM copy of sciencemadness no more than a few months out of date. The preparation is mostly automated. It just takes a while to download and upload a few gigabytes over my home internet connection.

quicksilver - 25-12-2010 at 10:32

Polverone , thank you for the work and time (Merry Christmas, too)

I am NOT of a paranoid mind-set but if I lived in certain countries that developed a precedent of filtering content or whatever......I think it might be a damn good idea to download a copy & keep it on DVD.


Edit:
-{I'm glad I got back in time to edit this.}-

I had some questions and I am glad to report some answers so that if any folks have similar questions they can make use of some experimenting I've done.
VMPLAYER (v2.x from 2008) is what you want if you want to run your backup on a old windoz machine. I have two machine running SUSE (a little one with 9x with very little processing power and a larger one with a great deal more RAM and processor. IF you want to run a VM on a machine without Linux but it's fairly new and strong - no problems. But IF you have an old laptop with 1Gb RAM and a CPU that has something like 1 GHz the old one will function even under some older version of windoz. So VMPLAYER (v2.x from 2008) is what you would likely be doing best at using as it's VERY limited but it has VERY low demands.

[Edited on 26-12-2010 by quicksilver]

woelen - 28-12-2010 at 04:09

I revived the website, including the scipics and the protected forums on a dedicated server, with Ubuntu, Apache, PHP and MySQL installed and I have had it all running and tested by Polverone through the internet on my own domain. After Polverone's successful testing I brought it down again. So, the instructions of Polverone are perfectly suitable to revive the website on a new domain and new hardware if that ever may become necessary.

TheOrbit - 10-1-2011 at 03:28

i downloaded zip file several times but it keep giving an error

Quote:

sm_offline.zip: The archive is either in unknown format or damaged


even when i check to leave broken files it doesn't work ???



Polverone - 10-1-2011 at 07:23

Quote: Originally posted by TheOrbit  
i downloaded zip file several times but it keep giving an error

Quote:

sm_offline.zip: The archive is either in unknown format or damaged


even when i check to leave broken files it doesn't work ???



The archive should be 3728725636 bytes in size and have the MD5 hash 50afa10d0373b020f2f0a44b8480d722. If your copy does not match those parameters, it has not been downloaded correctly. If it does match those parameters, I suspect something wrong with your computer (bad RAM or bad archival program). The archive has been successfully downloaded and unpacked before.

Remember to use a client with support for resuming downloads. The host the file is on is inexpensive, but touchy with very large files. You will almost certainly get an incomplete archive if you just use the stock download option in IE or Firefox.

woelen - 10-1-2011 at 11:20

Use the program http://users.ugent.be/~bpuype/wget/ for downloading this file. This has worked for me very well. Just start it in a shell (or DOS box if you are on Windows) and leave the computer running for a few hours.

dann2 - 10-1-2011 at 12:18



Use the -c switch
eg: (at command prompt)...> wget -c http://XXXXXXXXXXXX

TheOrbit - 10-1-2011 at 14:09

thanks for help, i was able to extract it on my mini rather than the desktop i used first " that is weird :o as the desktop is of higher features", thanks . :)

dann2 - 10-1-2011 at 19:39

Hello,
Can you run the offline version of SCI.MADNESS.ORG with your internet connection disconnected (pull the actual wire out of the back (or side) of your computer).
My 'offline' version works OK but when I pull the internet connection from my computer it ceases to function. Not very off line!

Dann2

woelen - 10-1-2011 at 23:31

Yes, you can have the offline version running, but then your system needs a working local host. I know of some computer systems, which have no working TCP/IP stack when there is no physical network interface on the system. Other computer systems always have a loopback device with IP-address 127.0.0.1 which always is there, even if no physical network interface is present. Check whether you can connect to 127.0.0.1 (or localhost).

TheOrbit - 11-1-2011 at 04:01

sorry, i have another question. i export the database of the forum " about 500 MB" from phpmyadmin then i tried to import it in my local xmb forum " using both bigdump and mysqldumper" but it restore single 1st table "attachments" , and keep give errors f mysql.
is there is any way to import this big file into local forum ????

Polverone - 11-1-2011 at 16:46

Quote: Originally posted by dann2  
Hello,
Can you run the offline version of SCI.MADNESS.ORG with your internet connection disconnected (pull the actual wire out of the back (or side) of your computer).
My 'offline' version works OK but when I pull the internet connection from my computer it ceases to function. Not very off line!

Dann2


I understand that this problem comes up when DHCP is not available for the virtual machine to obtain a network address from. It's DHCP service from your router that is important, rather than a live internet connection: unplug the external internet connection from your router and the backup will still work fine. Last year I tried a couple of experiments to get a DHCP service running on a completely unplugged machine, just for host-guest communication, but was unsuccessful. Until someone who knows networks better can advise, I don't know how to solve it.

Polverone - 11-1-2011 at 16:53

Quote: Originally posted by TheOrbit  
sorry, i have another question. i export the database of the forum " about 500 MB" from phpmyadmin then i tried to import it in my local xmb forum " using both bigdump and mysqldumper" but it restore single 1st table "attachments" , and keep give errors f mysql.
is there is any way to import this big file into local forum ????


If you use mysqldump/mysql from the command line it should work fine. I know that I have had problems using graphical, web-based dump/restore tools with the forum before. The forum with attachments included is much larger than 500 megabytes. I had to stop using phpmyadmin for export years ago because it seemed to stop working as the database got large.

[Edited on 1-12-2011 by Polverone]

Sciencemadness torrent

slinky - 12-1-2011 at 00:14

Complete Sciencemadness backup hosted on a 1 gbit line.

------------------------------------------------------------------------------

sciencemadness.org backup from 12.24.2010

contents:
1.1G library.zip
3.3G los.alamos.technical.reports.zip
1.7M member.publications.zip
4.0K sciencemadness.org.backup.nfo
4.1M sciencemadness.site.backup.pdf
89M scipics.zip
3.5G sm_offline.zip

Notes:
The forum backup itself is the same as from this thread. This torrent also includes the content from the rest of the site.

------------------------------------------------------------------------------


Attachment: sciencemaddness.org.backup.torrent (40kB)
This file has been downloaded 1548 times


[Edited on 1.12.2011 by slinky]

The very same data as the last torrent.

slinky - 10-3-2011 at 14:46

Only the torrent name is spelled correctly, typo free :P

Attachment: sciencemadness.org.backup.torrent (627kB)
This file has been downloaded 1599 times

backup updated

Polverone - 5-4-2011 at 09:29

I have updated the forum backup.

The new backup is 4065379728 bytes and has MD5 sum 390171a5f7eb650d7ca8bd18e688cbd3.

Background and instructions for use are in my previous "sciencemadness contingency plan" post.

AndersHoveland - 23-5-2011 at 15:30

I think a archived backup for science madness is very important. This forum contains some of the best chemistry information for amateur experimenters available. This forum will not last forever, and it would be very unfortunate if the information contained within became lost and unavailable to online readers ten years from now.

Many people now lament everything that was lost in the Explosives and Weapons Forum, and even The Hive.
The few records left of the posts made in those forums are now much-sought after by many online readers. I think there should be at least five people that should save full records from this forum, and then come back 5-10 years from now to reintroduce these records back to the online world.

That being said, I think some effort should be done to weed out all the unimportant topics when saving back up records.
This could substantially reduce the file size, making downloads much faster. It would also save future readers from having to sift through so much trash.

Also the search function for this forum is horrible. Even doing a google search, "sodium hydrazide science madness" failed to find the post I made about this unusual compound, although typing "science madness NaN2H3" does find it. Hopefully it will be easier for future readers to search for what they are looking for in the forum records.

Also, when you include a link, please make a quick summary of what important information you found in the link. Many of the links in older posts no longer work. If you take pictures, it would may be helpful to write a short description of what is in the picture, as picture files may not make their way into all versions of archived forum records.

[Edited on 23-5-2011 by AndersHoveland]

Eliteforum - 20-6-2011 at 03:27

Is the offline backup download speed limited? I'm on a 50mb/s pipe and I'm getting roughly 50 to 80kbps. It's going to take at least 10 hours to download at the current speeds.

Edit -

Currently trying to grab it, but it's taking it's sweet ass time :)
Quote:

>wget http://library.sciencemadness.org/archive/sm_offline.zip

SYSTEM_WGETRC = /wget/etc/wgetrc syswgetrc = /wgetrc --2011-06-20 12:17:43-- http://library.sciencemadness.org/archive/sm_offline.zip

Resolving library.sciencemadness.org...

67.23.244.212 Connecting to library.sciencemadness.org|67.23.244.212|:80... connected.

HTTP request sent, awaiting response... 200 OK Length: 4065379728 (3.8G) [application/zip]

Saving to: `sm_offline.zip' 7% [==> ] 320,536,929 71.4K/s in 60m 53s 2011-06-20 13:18:37 (85.7 KB/s) -

Connection closed at byte 320536929. Retrying. --2011-06-20 13:18:38-- (try: 2) http://library.sciencemadness.org/archive/sm_ offline.zip

Connecting to library.sciencemadness.org|67.23.244.212|:80... connected. HTTP request sent, awaiting response... 206 Partial Content Length: 4065379728 (3.8G), 3744842799 (3.5G) remaining [application/zip] Saving to: `sm_offline.zip' 15% [+++==> ] 648,598,594 77.3K/s eta 7h 41m



[Edited on 20-6-2011 by Eliteforum]

[Edited on 20-6-2011 by Eliteforum]

Polverone - 20-6-2011 at 09:45

It's not deliberately speed limited, but it is sitting on a cheap host. They don't meter bandwidth use but that also means they have little incentive to provide more bandwidth.

Eliteforum - 20-6-2011 at 10:49

No worries, I'll get there in the end!

Eliteforum - 22-6-2011 at 08:36

Finally got it.

Althernative download link at..

http://www.criminally-inclined.com/viewtopic.php?f=12&t=...

KemiRockarFett - 11-8-2011 at 11:21

Quote: Originally posted by Eliteforum  
Finally got it.

Althernative download link at..

http://www.criminally-inclined.com/viewtopic.php?f=12&t=...



Perfekt I am taking it down right at the moment.
As Anders Hoveland wrote. Its importat to get backups.
Big brother regulates everything right now because of the so called terrorist. Probably its just a question of time until information at forums will be canceled by the big fat big brorhter EUSSR government.

AndersHoveland - 21-9-2011 at 17:56

When was the last time a file back-up of this forum was made?

Does an archive of Science Madness exist, in the event that this forum was permanently shut down? If so, does this archive also contain all the picture files and images? Some of the threads will be difficult to understand without the molecular diagrams shown in the images.

We can look at what remains from the E+W forum. While there are a few archives from that forum still floating around, they are difficult to find, and it appears that much of the information was lost. Science Madness should prepare now for the future.

I would hate to see half the posts in this forum lost ten years from now.


backup updated

Polverone - 10-10-2011 at 18:38

I have updated the forum backup.

The new backup is 4566234886 bytes and has MD5 sum 178b6e9d360381b87e8398ef09c1d6d1.

Background and instructions for use are in my previous "sciencemadness contingency plan" post.

backup updated

Polverone - 22-1-2012 at 16:20

I used the forum downtime on January 18 to prepare a new backup.

The new backup is 4791775448 bytes and has MD5 sum 82e30e3b537118aaa285fe094c459f7b.

Background and instructions for use are in my previous "sciencemadness contingency plan" post.

Aqua_Fortis_100% - 22-1-2012 at 19:30

Quote: Originally posted by AndersHoveland  

We can look at what remains from the E+W forum. While there are a few archives from that forum still floating around, they are difficult to find, and it appears that much of the information was lost. Science Madness should prepare now for the future.

I would hate to see half the posts in this forum lost ten years from now.


Yeah, it was difficult to find a decent torrent archive and yet was very frustrating compared with the good/old E&W board. The domain roguesci . org is basically a live museum now, dead.

Many thanks to people here and adm himself (Polverone) initiatives to have an active backup on SMDB.

Now with pigs trying to pass this SOPA/PIPA threat, means of legitimate forum defense and the continuity of science discussion are (more than before) vital. Backup is one of these ways.

This is my favourite forum and it cant be vanished in anyway. NEVER!

Long life to SMDB!

Eliteforum - 15-2-2012 at 04:18

Once I've my fibre optic broadband installed on Saturday (18/02) I'll download the latest version and host it for alt. download.

Eliteforum - 18-2-2012 at 06:02

You may want to edit this part of your initial post:
Quote: Originally posted by Polverone  

It is big, about 1.8 GB. A successful download should have the MD5 hash 6a41b7a28c999a243a388f6db5d83d62.


To reflect the current file size and hash tag.

I'm currently downloading the latest version and shall offer it up via my website as an alt. download.

Eliteforum - 19-2-2012 at 06:59

Latest now available.

http://criminally-inclined.com/viewtopic.php?f=12&t=58&a...

http://criminally-inclined.com/download/sm_offline.zip

quicksilver - 19-2-2012 at 08:21

Quote: Originally posted by Aqua_Fortis_100%  

Yeah, it was difficult to find a decent torrent archive and yet was very frustrating compared with the good/old E&W board. The domain roguesci . org is basically a live museum now, dead.

Many thanks to people here and adm himself (Polverone) initiatives to have an active backup on SMDB.

Now with pigs trying to pass this SOPA/PIPA threat, means of legitimate forum defense and the continuity of science discussion are (more than before) vital. Backup is one of these ways.

This is my favourite forum and it cant be vanished in anyway. NEVER!

Long life to SMDB!


Actually, there was an exceptionally in-depth backup of the E&W Forum that was surprisingly complete. If I am correct it was posted about two or three months before the actual E&W Form really disappeared. It actually took up more space than the backup design for SciMad but that may have been due to simply a different style of backup. but everything was there.
It was only on PirateBay for about sixty days then it was gone. Another one was posed but that was just the HTTP copy. I think even that's gone also.
The backup concept for SciMad was a good idea as it runs independently. The one I saw a couple months back was good.

ethicalhacker - 25-2-2012 at 14:48

After working with Polverone to resolve an issue I had with the official site backup, the following notes emerged.

1) Windows 7's integrated zip handler is unable to extract the site backup zip file (complains about needing 734PB of free disk space). Extracting with a 3rd party program resolves this issue.

2) Most every VMDK reader/writer/converter that runs on Windows will not be able to use the VMDK file contained within the site backup. If one wishes to use this VMDK in a non-prescribed manner, one must first import it into Virtual Box. The import process creates a mostly copied but slightly modified VMDK. One can then use the imported VMDK for various non-standard purposes.

Polverone tells me that the VMDK is generated on a Linux box, so my theory is that the VMDK contained within the site backup has some Linux file headers. Naturally, most Windows applications don't take that into account. Virtual Box, being a cross-platform application, probably makes the necessary modifications during the import process.

Eliteforum - 26-2-2012 at 13:13

I have no issues using W7's own zip tool (W7 Ultimate)

Polverone - 5-6-2012 at 17:53

Wizzard is now seeding the backup: http://www.sciencemadness.org/scipics/Science%20Madness%20Fo...

From now on backups will typically be distributed via bittorrent Note also that this backup, dating from early May, is in .ova Open Virtualization Format. I appreciate any members who are willing to continue seeding.

Rosco Bodine - 6-6-2012 at 14:37

What is the tracker ? I get an error message on that.

Wizzard - 7-6-2012 at 13:42

No idea what's causing the issue, but the torrent seems fine.

Trackers are openbittorrent and publicbt... I have 2 other people downloading now :)

Rosco Bodine - 7-6-2012 at 18:39

Everthting is working okay now. Upgraded old P2P client to Bit Torrent. I was using the old Bit Tornado and it wouldn't cooperate.

madcedar - 20-9-2012 at 07:51

Is it possible to do rsync style incremental backup every four or six months?

woelen - 23-2-2013 at 14:04

I tried to download the sciencemadness backup just to have another place where it can be stored, but I obtain the following error:

error: Problem connecting to tracker - ('url error', 'unknown url type', 'udp', 'udp://tracker.publicbt.com:80/announce?info_hash=%E91r%9Ct%AC%D7%03%DB%FA%18%04%60s%F5%83%B5J%B3%88&peer_id=T03I-----khTYUeTo0ZU&supportcrypt o=1&port=49181&uploaded=0&downloaded=0&left=4817457152&no_peer_id=1&compact=1&event=started&key=idEM3D')

I use torrentflux for downloading torrents and that program works fine with other torrents.

AndersHoveland - 24-2-2013 at 23:36

Is it possible to place a priority on some threads, so that not every last thread has to backed up?
You could make available two back up copies: one containing all the threads, and another that does not contain the not so important threads, to reduce storage space.

Eliteforum - 26-2-2013 at 01:52

That would require a fair bit of time consuming work, and what you find to be a priority, someone else may not. It works just fine as it is.

Polverone - 4-3-2013 at 10:56

The first new backup in several months should be available in a few hours. It is now about 5.3 GB. U2U me if you can seed a torrent or host a copy on web/ftp space. I can provide copies to a few people directly from the web site here, but it will take too much bandwidth to allow everyone direct http download.

Eliteforum - 5-3-2013 at 02:56

As per the u2u, I can host the backup if required.

Polverone - 29-3-2013 at 19:21

This backup is current to the beginning of March. It is 5,221,074,432 bytes, md5 hash 44518ee04178bd2ba3ea4971d6473516.

http://www.sciencemadness.org/archive/SM-offline2.ova

This is a format designed for use with VirtualBox.

I don't actually need third party hosting, though I would still welcome it, since the provider I use for this site has just increased the monthly bandwidth allocation by 10 times.

Polverone - 16-5-2013 at 23:33

This backup is current to May 15. It is 5,428,217,856 bytes, md5 hash 6c73f96175a56cd0938823a751b68fb7.

http://www.sciencemadness.org/archive/SM-offline2.ova

This is a format designed for use with VirtualBox.

Polverone - 3-8-2013 at 19:23

This backup is current to August 2. It is 5,635,886,080 bytes, md5 hash 1bc1d2f46cc9a28f3982dce3b8059fa4.

https://www.sciencemadness.org/archive/SM-offline2.ova

This is a format designed for use with VirtualBox.

Polverone - 13-9-2013 at 10:39

This backup is current to September 10. It is 5,726,903,808 bytes, md5 hash e71f082660c3ba27a246ced1d147540f.

https://www.sciencemadness.org/archive/SM-offline2.ova

This is a format designed for use with VirtualBox.

Backup Torrent [Finally] Online

bfesser - 26-9-2013 at 10:54

Quote: Originally posted by Polverone  
This backup is current to September 10. It is 5,726,903,808 bytes, md5 hash e71f082660c3ba27a246ced1d147540f.

https://www.sciencemadness.org/archive/SM-offline2.ova

This is a format designed for use with VirtualBox.
I've created a torrent of this backup and set up a dedicated Raspberry Pi seed.

Attachment: SM-offline2.ova.torrent (14kB)
This file has been downloaded 1343 times

Trackers:
udp://tracker.openbittorrent.com:80/announce
udp://tracker.publicbt.com:80/announce

I hope this helps.

[update] I've since disconnected the RasPi to use for other tasks, as no activity was observed. If anyone wants it, U2U me, and I'll seed again for a while.

[Edited on 14.10.13 by bfesser]

User64 - 19-10-2013 at 12:06

Couldn't get the offline portion to function with VirtualBox but did get it to work with VMWare and one notice to lax hardware profile standards.

Polverone - 9-2-2014 at 16:23

This backup is current to February 7. It is 6,281,758,720 bytes, md5 hash 0da7ac78076d0c965c768139533806d6.

https://www.sciencemadness.org/archive/SM-offline2.ova

This is a format designed for use with VirtualBox.

Polverone - 2-6-2014 at 00:34

This backup is current to May 29. It is 6,509,149,184 bytes, md5 hash 9767d1240af4cce0659710c4c87fd475.

https://www.sciencemadness.org/archive/SM-offline2.ova

This is a format designed for use with VirtualBox.

improved drop-in search for vm archive

food - 14-11-2014 at 15:40

Hi. Firstly a big thanks to Polverone, and anyone else concerned, for making available the offline archive.

tl;dr The attached file contains everything needed to easily and greatly enhance the search in the offline archive.

I was looking at the search form and code in the offline archive. The search is very poor. While searchers of the live, online, site can be pointed at google for site specific searches the offline nature of the archive rules this out. I was looking at trying to improve the existing sql when I came across the Sphinx open source search server project and decided to give it a go. One thing led to another, and the result is the attached file. The version of Sphinx that would be available through the vm package management software would be version 0.9.8, which is probably getting out of date. I've compiled the current version of sphinx, 2.2.5, and made a .deb file for easy installation.

Unpacking the attached file in the vm and running the INSTALL_SPHINX.sh will install Sphinx, create a Sphinx index, add an alternate form page and some associated database entries, and include Sphinx in the startup process.

If you use the vm at all I'd encourage you try this out. The addition of Sphinx in no way alters the existing functionality; the new search is performed from an added php form. If the new search turns out to be buggy, and I'm not seeing that yet, then you need only revert to using the old one. Those who are interested can easily tweak the added search via the code in the php form, and/or reindexing with new parameters.

Disclaimer: I had to resolve different bugs as I shoehorned this thing in. I think that it's good now, but perhaps it may be considered a beta version.

Attachment: sphinx-2.2.5-scimad.tgz (7.5MB)
This file has been downloaded 1345 times

[Edited on 14-11-2014 by food]

Polverone - 19-12-2014 at 13:51

The archive has been updated. It is now current to 12 December 2014. This update was delayed longer than usual because the archive has grown in size such that I had to resize the underlying virtual disk image before I could produce another release.

Food, that looks like an excellent contribution and I'm sorry I missed seeing it before I prepared this release. I will try it and include it in the next release if it looks to be stable.

https://www.sciencemadness.org/archive/SM-offline2.ova

This release is 8,108,819,456 bytes in size, md5 hash a069264c7ca792f37054870a92061bec.

Polverone - 12-5-2015 at 15:37

Archive updated again. Unfortunately forgot to revisit this thread before preparing it so the search contribution still hasn't been added. Archive is now current through 6 May 2015.

http://www.sciencemadness.org/archive/SM-offline2.ova

This release is 9,099,728,384 bytes in size, md5 hash 9f273ff2624d89c845ab02f7a210bcb3.

Polverone - 27-9-2015 at 11:59

Archive updated again. Archive is now current through 23 September 2015.

http://www.sciencemadness.org/archive/SM-offline2.ova

This release is 10,673,768,448 bytes in size, md5 hash accd4811ce4f8b8125b19793c3794f25.

The live forum is growing so large, and the archive itself is taking so much space, that I may have to take the archive offline in a few days just to free up space for the live forum. Please send me a U2U if you can host large files like this.

Polverone - 29-1-2016 at 16:37

Archive updated again. Archive is now current through 16 January 2016.

http://www.sciencemadness.org/archive/SM-offline2.ova

This release is 11,788,337,664 bytes in size, md5 hash 2371a99d11c848860b18933c37942233.

Remember, if you are unhappy with the administration of the forum, the archive is a chance to replicate Sciencemadness elesewhere, under different leaders and rules. The archive comes with tools to ease replication onto a new host, though you will still need reasonable familiarity with Linux/Apache/MySQL/PHP. This is a chance at <a href="https://en.wikipedia.org/wiki/Exit,_Voice,_and_Loyalty">exit</a>, so that I do not have the temptation or the means to leave members stuck in an organization they are unhappy with.

Polverone - 16-6-2016 at 20:41

Archive updated again. Archive is now current through 13 June 2016.

http://www.sciencemadness.org/archive/SM-offline2.ova

This release is 13,077,444,608 bytes in size, md5 hash 0fb52f91cac5efacb570deefb1cbef94.

gdflp - 18-6-2016 at 20:11

While the current backup is offline, if anyone wants it right now : https://drive.google.com/open?id=0B6GYzE-Eq2NmaVJFbVpKNUV0Nl...

Obstbraten - 28-2-2017 at 23:05

Is there a 2017 backup available? You could host it at google drive or mega.nz

Mabus - 15-5-2017 at 10:54

So... are there any backups of the site for this year?

Melgar - 19-2-2018 at 10:17

Okay, in case anyone is having trouble with this or wants to do something similar to what I'm doing, I got the virtual machine running remotely in a headless configuration. VirtualBox can be installed and run via a command line, making it a million times easier to work with via a standard terminal, rather than try to figure out all the quirks of the virtual machine terminal. This allows for easily accessing the guest's filesystem from the host, since you can use tools like rsync and scp over ssh. Since I didn't want to waste my phone's data, I spun up a new Ubuntu server on AWS, then downloaded gflip's copy of the virtual machine here:

https://drive.google.com/uc?id=0B6GYzE-Eq2NmaVJFbVpKNUV0NlE&...

The following perl script will be able to download the file from the above url:

https://github.com/circulosmeos/gdown.pl

Then I set it up so that I could SSH into it locally, using port forwarding as described here:

https://leemendelowitz.github.io/blog/ubuntu-server-virtualb...

The SQL database for the site, compressed, is about 8GB. This seems excessive. Will need to look into it some more.

Polverone - 9-6-2018 at 11:31

I have updated the archive a couple of times since I last posted about an update, but I haven't kept up with the announcements.

The archive has been updated again. It is now current through 2 June 2018.

http://www.sciencemadness.org/archive/SM-offline2.ova

This release is 17,826,357,248 bytes in size, md5 hash 17a2410fbb29563c776a81a04e70433c.

streety - 9-6-2018 at 11:53

I'm getting a "Forbidden" error on that file.

[Edited on 9-6-2018 by streety]

Polverone - 9-6-2018 at 12:10

File permissions have been fixed.

streety - 9-6-2018 at 12:20

The download has started so looks like it is now good.

I downloaded the previous backup and managed to get it running. Remarkably simple.

I downloaded it with the intention of helping Melgar migrate the forum software. phpBB is complex!

I may also parse the posts to extract any youtube links in my ongoing efforts to make copies of all relevant channels.

headrx - 7-7-2019 at 12:21

Anyone able to add this to a torrent? Us slower connections wont be able keep a solid connection long enoguh to get it

Polverone - 15-2-2020 at 11:39

The archive has been updated again, after a long delay. It is now current through 8 February 2020.

https://www.sciencemadness.org/archive/SM-offline2.ova

This release is 21,241,060,864 bytes in size, sha256 hash aa4f54ea040b4c826597accec11d20020fa686707a85c3a673c357a8474aa176.

[Edited on 2-16-2020 by Polverone]

Rosco Bodine - 15-2-2020 at 21:40

Quote: Originally posted by Polverone  
The archive has been updated again, after a long delay. It is now current through 8 February 2019


You meant 8 February *2020* ....NOT... 2019.... Correct ?

Also please confirm this linked VM is still valid

https://www.virtualbox.org/wiki/Downloads

[Edited on 2/16/2020 by Rosco Bodine]

fusso - 16-2-2020 at 04:06

So when will we move to new forum software?????:mad:

[Edited on 200216 by fusso]

woelen - 16-2-2020 at 11:48

Quote: Originally posted by fusso  
So when will we move to new forum software?????:mad:

Why such an angry message? Keep in mind that this is all voluntary and unpaid work from the side of Polverone. Of course you can ask for when new software will be available, but sending such an angry-looking message is not the way to go. Also keep in mind, that despite all kinds of problems with software and otherwise, the forums still are alive and kickin' with an active and enthusiastic member base, already for 17 years! For this I want to thank Polverone and of course the community around sciencemadness! How many other forums survive that long? Most of them go out slowly and silently, some after just a few months, others after a few years.

Polverone - 16-2-2020 at 12:15

Quote: Originally posted by Rosco Bodine  
Quote: Originally posted by Polverone  
The archive has been updated again, after a long delay. It is now current through 8 February 2019


You meant 8 February *2020* ....NOT... 2019.... Correct ?

Also please confirm this linked VM is still valid

https://www.virtualbox.org/wiki/Downloads



Oops! Yes, 2020. Message updated. The VirtualBox virtualization software is still the right choice for running this image.

RogueRose - 8-2-2021 at 17:18

Quote: Originally posted by Polverone  


Oops! Yes, 2020. Message updated. The VirtualBox virtualization software is still the right choice for running this image.



Is it possible to do a differential or incremental backup? I think it would be great if every year there was a full backup made and then do an incremental or differential backup each month so people could just download the monthly updates to have their backup fully functional with the latest posts. It's just a thought, I really hate where I see some things going with online censorship and this is one area where being proactive is priceless.

If you need or want any help doing this or figuring out if this is possible, I'd be happy to do so. I'll also be happy to host any/all of the files on a HTTP/FTP download page and as torrents.

Polverone - 7-8-2021 at 12:59

The archive has been updated again. It is now current through 25 July 2021.

https://www.sciencemadness.org/archive/SM-offline2.ova

This release is 25,143,577,600 bytes in size, md5 hash dc067d28948ed16ca36a73e6cfa7d3c6.

RogueRose - 13-10-2024 at 15:26

I was wondering if there is a chance of updating the site backup or if there is a way to create a differential file that could include any new threads since the last backup or ideally any thread that has been altered since the last backup.

I would be willing to host any file (the original, a differential or a new complete backup) on my torrent servers as I have 2, one that is 10GBps & one that is 1Gbps.

The only issues I can see about doing an differential is it might be difficult to incorperate it into the existing files unless the files just over-write the older files with the updated files.