Importing the old data

[Thamus_Knoward]

Hi @geeberry and @illblew,

I'd like to help with the import process where I can, but in the age we live in I am sceptical that importing old threads by hand is the only solution. Could you give us an update on where we're at and what the problems are?

I supposed you are aware of this, right?: https://xenforo.com/help/importing/

The last info I have regarding the export is that you used this exporter:
https://github.com/spikeh/enjinuity

Is that correct? If so it should be straightforward to import the output data from it using the native Xenforo imports.

Please give us some clarity on where this process isn't working out.

I am sure that if we stick our heads together we can find a solution that isn't a) error-prone, b) subjective and selective and c) hellishly time-consuming.

 

[Emoticone11]

Figuring this out should definitely be a high priority, since the more duplicate threads people transfer over in the interim the more resolution we'll also have to do by hand between the duplicates once the former threads are transferred over.

 

[geeberry]

Yes I am 100% aware of the importing section on Xenforo lol its the first thing we looked at? We haven't been lazy or uncaring about the data transfer...Me and will blew have tried and researched everything. Enjinuity doesn't work - its broken. Its over a year old and is basically a web scraper which is dubious the data would even come out correctly. Not sure if its open source and someone wants to take a crack at fixing it, but good luck as its over my head.

The tool we used is this one: https://github.com/AusTac/EnjinExportTool whihc exports thread data in .json format. But heres the problem - you say it should be easy to export to MyBB but Enjin PURPOSEFULLY makes it pretty much impossible to leave their ecosystem without leaving everything behind. We couldn't get email addresses for all our users and transferring users is useless without them. Again Enjin does this on purpose. the EnjinExportTool works but there is no .json importer made for Xenforo which is the standstill we are at.

If you guys have any ideas, post away!

 

[Emoticone11]

I'm not really that familiar with how MyBB and forum importing/exporting work, but how difficult would it be to write a parser to convert JSON to MyBB, which can then be used by the Xenforo import tool? (Assuming such a converter doesn't already exist)

 

[geeberry]

Im not sure on the difficulty - its over my head. We might be able to import the date into a wordpress, then import THAT data into xenforo

 

[Dan_Prime]

What format does Xenforo require? If you could link me to documentation or something I could take a look. The importing page that was linked earlier wasn't too specific on the technical details.

 

[geeberry]

I believe its .xml. No idea on documentation, only thing I can find is that import page. You have admin on the site Dan if you need to look through anything.

 

[Thamus_Knoward]

After installing Node.js and then PhantomJS, you can clone this branch on my fork of enjinuity, then you should be able to install it the way a package in python should install after navigating into the directory:

pip install -e .

Then using Python 3.6 it is possible to just import the package and use the functions.

import enjinuity

I am not too familiar with selenium or xcode queries so I'd appreciate any input with the JS side of things, put I think the parsers would work if we manage to identify the exact source of the data we need. Currently in enjinuity the xpath strings seem to be written to scrape an older version of Enjin. I think I do understand that now in the admin user list, enjin stores user data in some static data backend which is parsed and dynamically build clientside using AngularJS.

So using enjinuity might be one possible way of attack, the other would be to rewrite the data scraping part to access the Enjin API and making proper queries instead of trying to scrape it off the HTML. Here, UserAdmin.get seems to be the way to go.

@Dan_Prime what do you think?

 

[Dyvim]

Thamus,

Have you tried using the API UserAdmin.get to see if it returns email addresses with the user info? If so, that's pretty much the only way to get the email addresses from what I can see. (aside from a select few users who use their email as their username, haha)

 

[Dan_Prime]

@Dyvim is correct, if we use that API function we can retrieve all of the registered users. I'm not sure if it will provide emails, though.

Get all registered users on your website and specific data about each user. To retrieve one specific user,use the optional "user" parameter with the user's ID. Additionally, you can use the optional parameter

I could write a node script to retrieve all of the users most likely, the only thing is that I have no idea as to how the data needs to be reformatted. Xenforo has no official documentation and I haven't really been inclined to search for hours on my own. If anyone can point me to documentation on how the import data should be specifically formatted, I could take a look.

Another thing, I would need site admin on enjin. This method requires an API key and I can't generate one with my current permissions. Adittionally it would be useful to view the user data to validate what the API sends back.

 

[geeberry]

You literally can't get email addresses from Enjin. It was one of the first things we tried, including UserAdmin.get. They keep it 100% hidden, intentionally. I hope we missed something but getting users transferred is not gunna happen without emails. You can see what available data we could mostly get here: http://testwesteros.enjin.com/api/get-tags

@Dan_Prime I will send you an API key, and try and figure out how the import data should be formatted, which may help us with the threads.

 

[Thamus_Knoward]

Do we need a table with userid: email? If all else fails we could get all active builders, mods, editors to append their details to a google spreadsheet. Then we pool all users we don't have in a mock account or so?

This would allow us a partial salvation of the data, or not?

 

[Thamus_Knoward]

We don't really need all XX thousand users who signed up but never posted anything and are no longer active anyways.

 

[Emoticone11]

Surely importing the forum data is far more important to focus on than importing users? Or am I missing something where having user data is necessary for importing forum posts?

 

[Dyvim]

I'd assume that by having posts missing their users would cause some dependency issues on an import. And I don't know how the posts would display in Xenforo without having corresponding user data. Unless you just replace all instances of the users with a generic pre-existing user.

But it looks like a custom importer will have to be crafted either way. I didn't know that the emails were quite literally untouchable, I spent a lot of time trying to get to them today. (which was plenty more fun than actual work!)

 

[geeberry]

Yeah - i bugged the enjin admins about emails and they pretty much told us it wasn't happening. Another reason to dislike enjin Plus the fact that they make almost no improvements to their system the last two years but then all of a sudden I get spammed with their new digital bitcoin currency called enjincoin or something. Hard pass.

I will dedicate some time this weekend or the next week during work about the import format. If someone needs temp access to admin panel please message me and Ill promote you for a bit.

 

[Emoticone11]

So we have username info but not emails for users? Can we make a script that moves the username into the post contents, and then posts it as a generic user like Dyvim said?

e.g. an imported post would look like (brackets added to distinguish it from someone addressing another person in context)

[Emoticone11:]

blah blah blah...

 

[Dan_Prime]

They do have a point, if we import the old threads there will be posts made by users who do not exist. Would that even work, and do we want the forum cluttered with all that?

 

[geeberry]

Not sure if it would work, but I ran across a plugin for Xenforo that would allow us to change usernames of thread posters, so it could fix that easy I believe.

 

[Emoticone11]

I'm wondering at this point if it might be cleaner to create a subforum called "Archive" or something, which in turn contains copies of all the normal subforums, and then we move all of the threads from the old forum into Archive. Then we can selectively move threads out of Archive into the normal forum, which is a lot easier than manually copying the contents of the threads over.

By the time we figure out how to import, I feel like it's going to be super messy to transfer things over into the normal forums in an unstructured way.