Jump to content

Looking for help scraping data


Oddly

Recommended Posts

I'm scraping http://www.mtgtop8.com for data on magic the gathering decks. I will be using the data in a genetic algorithm to generate a deck for MTG based off pro statistics. The data scraping process takes a bit of time, so i'm looking for other machines to run this scrape script on so i can get data faster, anyone interested in helping me run a script for a few hours?
You must have python3, and i will send you the setup process and script via a pm.

Link to comment
Share on other sites

28 minutes ago, Damian666 said:

give me moneys, then ill do it xD

 

i feel like i have enough computers and friends that i won't need to pay for it. Otherwise i'd just rent a couple servers for 2 days

Link to comment
Share on other sites

8 minutes ago, Damian666 said:

no offence bro, but then why ask here?

 

just curious :P

cuz the more systems i can get the faster i can write the algorithms i need. right now it will take me a week to scrape all the data i need. 

I need to process 30,000 web pages, and its about 1 web page every 4-6 seconds.

Link to comment
Share on other sites

ok, but let me quote you:

9 minutes ago, Crest said:

i feel like i have enough computers and friends that i won't need to pay for it. Otherwise i'd just rent a couple servers for 2 days

 

im confused :P

Link to comment
Share on other sites

Just now, Damian666 said:

ok, but let me quote you:

 

im confused :P

I can do the entire process without paying for it on my own i i wanted to, the more systems i have the faster the collection process takes. I'm asking for people who have an extra system they're not using that they will let me run code on for a couple hours.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...