WebDataExtractor
This netorguyuschy expert is designed to convert the unstructured web pages of information in a structured tabular data. It downloads and parses the HTML file, it creates a hierarchy of the DOM (Document Object Model), and then applies these selectors CSS (Cascading Style Sheets, Eng.) For the required data fields and allocates them to specific values. In other words, it is – a powerful, customizable converter from HTML to CSV (Comma-Separated Values).
The expert can handle page (using WebRequest MetaTrader) or local files. To use the WebRequest make sure your domain is added to the appropriate experts preferences dialog.
The demo version of the expert and even the full version of running the strategy tester, can not use WebRequest due to limitations MetaTrader. The tester is available only to local files.
The expert would “read” the economic news, market reports and other useful information available on the Internet, but are difficult to analyze programs for free formatting and “decorations”, inserted in the page for human perception.
The tester demo expert randomly selects some fields and replaces them in the real data on the “demo” line. This is done intentionally to prevent the possibility of a full conversion demo data.
Options
- URL – web address (beginning with http: // or https: //), or a local file name (must be in HTML format);
- SaveName – the output file (CSV);
- TimerSeconds – frequency of restarting the URL and its treatment; 0 means that the process is performed only one time and then unload itself expert;
- TriggerVariable – optional global variable to check at the time of the timer; If specified, an expert checks to see if it: if so – EA starts processing, and then removes the variable if no – EA misses the call timer and waits for the next;
- RowSelector – CSS selector to highlight data tables;
- ColumnSettingsFile – a CSV file with the selectors for the speaker (see description in the comments.)
- SubstitutionSettingsFile – optional file in the CSV format from substituting rules (see description in the comments.);
- TestQuery – test CSS selector for the rows of the table;
- TestSubQuery – CSS test selector to the column in a row;
- LogXXX – enable / disable logging XXX option;
Operating principle
To highlight the data necessary to examine the source code (HTML) of the web page. This can be done in any modern browser with built-in web developer tools. For example, in Chrome you can open the “Developer tools” by clicking “Settings” button (in the upper right corner of the main window), and then in the drop-down menu: “Tools” -> “Developer Tools” (the menu structure may vary from version to version of the browser). Details – in the official description of the developer tools – DevTools overview (Eng.). The most important for us is the section Inspecting the DOM (Eng.).
The user must find a page all its fields of interest and to compare them with sootstvuyuschimi DOM elements in HTML. On the basis of the characteristic attributes of these elements can form suitable CSS selectors that uniquely allocate a value for each data field. The developer has a command line window, where you can enter CSS selectors and check their work on the fly. Note, however, that the expert supports a limited set of selectors.
Another method of analysis assumes that you want to enable logging in the DOM Expert (LogDomElements) And explore MetaTrader Expert magazine. It is not as clearly and quickly, but does not require any external tools. To test the selectors may be administered in settings TestQuery and TestSubQuery. It is recommended to do it on a local file (URL It must be a file name, and not the address of the Internet). To download a web page to a local file, set the parameters as follows: URL – Address of the website, SaveName – file name for the record (if it exists, it is overwritten) RowSelector – empty.
Supported CSS selectors
For details, refer to the discussion.
examples
Examples are also given in the discussion.
Related posts
-
Interactive Stop Loss Demo Demo version Adviser Interactive Stop Loss. This version is specifically designed for visual backtesting with a tester…
Forex Trading 