Public endpoint to 2k/ip/hr (previously 1k)
Oauth endpoint to 20k/ip/hr (previously 10k)

Per application limit (identified by your Access Key): 100,000 calls per day will remain the same.

With this increase you can get up to 3 million – 3.1 million requests per month. (100k cap per day)

Thank you to the YQL community – we hope you enjoy this increase.

Console:

• New debug checkbox.
  • Checking this will simply add debug=true to your console URL. When debug=true is set, it enables open table debugging and viewing of YQL network calls.
• New expand checkbox.
  • Checking this will expand the results section to full height.
• Renamed “My Tables” to “My YQL”.
  • With this release, you have the ability to create YQL tables, environments and js files via the editor. The My YQL section lists those files based on those types. This is the section where you can launch the editor to edit the files and the area where you can delete your files.
    
    

    Some actions when clicking on the file names:

    • Clicking on a table name will put the store execute key into the yql statement area and desc the table.
    • Clicking on an environment name will load that environment by adding env=store://(your store execute key here) to the console.
    • JS files are only editable via the YQL editor.

    Only you will know what YQL files you have created, since you need to be logged in to create and view them. But you can share your YQL file store execute keys, as they can be run by anyone after sharing. Only share them if you want others to run that file.

Editor:

• The ability to create different YQL file types.
  • You now can create Tables, YQL environments and JS files. Simply select the “Save as” drop down to save the type of file needed.
  • Regarding JS files, these are files you can y.include() into your execute statement in a table.

• Dragging a YQL file from the sidebar onto the editor will produce a contextual code snippet.

  

  • Saving file as: Table or JS
    • Dragging a Table file will produce: y.use("store://execute key here","namespace here");
    • Dragging a Environment file will produce: y.env("store://execute key here");
    • Dragging a JS file will produce: y.include("store://select key here");
  • Saving file as: Environment
    • Dragging a Table file will produce: use "store://execute key here" as name_space;
    • Dragging a Environment file will produce: env "store://execute key here";
    • Dragging a JS file will produce:
set change_var_name="store://select key here" on change_to_table;
• Full screen layout
• Sidebar which contains Sample templates, keys of a table, and a list of your files by type.
• Changed file access FROM: tableid=id_here TO: id=id_here

For Example: select * from html where url=”http://finance.yahoo.com/q?s=yhoo” and compat=”html5″

Please start using this feature and give us your feedback! Eventually the new parser which supports ‘html5’ will be made default, but that will follow an announcement. Even after the new parser is made default, the old one can still be used by having compat equals to ‘html4’.

The YQL Editor is a simple and easy way to create your table in Yahoo’s cloud. The editor makes use of the yql.storage table to store your table with Yahoo’s cloud instead of hosting it on your own server.

Simply access it from the YQL console on the upper right hand column under “My Tables”.

Some quick notes: You must be logged in to view, create and edit your tables. You can not view other people’s tables. If you previously created and stored tables using yql.storage, they will not show up in “My Tables”. Storing tables directly using the yql.storage table will also not be shown in “My Tables”. “My Tables” makes it easy to track tables you created via the YQL Editor while logged in.

Click the “new” link to launch the YQL Editor. This will open up the editor in a separate page. By default new tables are named “untitled_table”. You can rename the table by simply clicking on the name.

The Tables dropdown provides sample templates to construct your table. It also will show your tables if you have any.

When in the “My Tables” section in the console, clicking on the table name will put the store execute key into the yql statement area and desc the table.

To query your table store, put the yql query statment after the “use” declaration. For example: use “store://Tdr13p0ubxczYZ78ia0Sph” as zillow; select * from zillow where address = “1835 73rd Ave NE” and citystatezip = “98039″ and zwsid = “X1-ZWz1cse68iatcb_13bwv”

Quick note: You can share your table execute store, it can be run by the public. Your table store execute key is only known to you – unless you choose to share it.

You can also make your endpoint (which will be really long) into a query alias. Click on the “Create Query Alias” link on the top right hand side of the YQL statement box to customize your endpoint.

Currently, it can take up to 30 seconds to see changes made to your table after editing. This will be fixed in a future YQL release. By adding debug=true to your query (or console), you can see real time edits after saving.

We plan to add new features to the YQL Editor as time goes on. Future releases will include the ability to manage and create your own YQL environments and hosted Javascript files. Please let us know of any features you’d like to see at yql-questions (at) yahoo-inc.com.

YQL Table Health is intended to provide a quick general overview of how “healthy” the community OpenData tables are:

When you first arrive to the page, you will see a list of all the tables that can be used by YQL. Clicking on one of the entries in the list will cause it to expand and show additional information regarding where the source of the XML file is, what kind of table it is, sample query information, and lastly any errors that were encountered. You can use the controls on the left-hand side to further filter, sort, and search through all this data. If you see a table that doesn’t work, you can contact the author of the table to fix it via github. You may also fork the yql-tables from github and fix or enhance the table yourself.

YQL Table Health uses a sever-side script to iterate through all the tables. Each table’s XML file is loaded into YQL cloud storage before a series of checks are run against that XML file. The test results are then cached in a database as well as memory to serve this data as fast as possible. Updates to caches are triggered by users visiting the page and only fire if the data is older than thirty minutes; requests for an update are also synchronized to prevent a race condition occurring, where two or more requests might be made simultaneously. Last but not least, the data is served to the user through a user-interface built using HTML5 and the YUI Library.

The next tool we’re going to introduce to you is YQL Lint.

YQL Lint is essentially an XML debugger for individual YQL tables. You can enter either a URL to a XML file, or the contents of an XML file, and it will validate this against our schema for syntax flaws. Once the schema check has been passed, we will use YQL to get a description of your YQL table and check to see if it contains a sample query that returns a valid result. YQL Lint essentially relies on the same core backend as YQL Table Health.

Please experiment with these tools and send us any questions or suggestions you might have.

Daniel Park – YQL/Pipes Intern

For those of you relying on those tables please consider using the community BOSS v2 table (https://github.com/yql/yql-tables/blob/master/boss/boss.search.xml).

Thanks -YQL Team

The latest YQL release adds support for Comet-based streaming with Downstream Polling (”CDP”), which allows YQL clients to receive updates to their queries in real time.

Motivation

In traditional YQL, a client must poll the YQL server for updates, by sending the same YQL query over and over again. Each time, the YQL server parses the query into a Pipe object, executes it, sends the results to the client, and closes the response.

Traditional way of invoking a Pipe

This approach is inefficient and does not scale well for updates: First of all, the YQL server has to parse the same YQL statements into corresponding Pipe objects over and over again, where each Pipe object is used to produce only a single response, after which it will be garbage-collected.  Secondly, there is no guarantee that the response data will have changed, resulting in unnecessary network traffic and wasted Pipe constructions and executions.

Long polling is not a solution either: It is impossible for the YQL server to know for how long to keep open a response, because it has no way of telling when new data has become available. As with busy polling, Pipe objects are not reused.

CDP attempts to address these deficiencies: In this mode, the client opens a single persistent connection to the server and sends the YQL query in the initial request. The YQL engine on the YQL server parses the query into a Pipe object, but instead of discarding the Pipe after a single execution and closing the response, it holds on to both the Pipe (turning it into a Standing Pipe), which allows it to execute the same query repeatedly over a period of time, and the Comet-enabled response, which allows it to send updated results to the client asynchronously and in real time.

Periodic invocation of Standing Pipe

Polling Frequency

In order to enable a table for CDP, its developer must specify the frequency (in seconds) that is appropriate for polling the table’s downstream web service for updates, using the new pollingFrequencySeconds table attribute.

If the YQL query is mapped to a single table, then the frequency with which the Standing Pipe will be executed is equal to the table’s pollingFrequencySeconds. If the YQL query is mapped to multiple tables, then the execution frequency of the Standing Pipe is set to the largest polling frequency of the tables involved, to increase the likelihood that each Standing Pipe execution will yield updated results.

Check out the YQL documentation for an example of how to enable a table for CDP.

Future Enhancements

A future enhancement will have the YQL engine participate in a truly event-driven, publish-subscribe (Bayeux) style notification system, where a table’s downstream service will be a named source of events, to which the YQL engine will subscribe through the appropriate event channel.

Implementation Status and Limitations

The current implementation of CDP is considered experimental and is made available on separate YQL web service endpoints, which are named after the traditional YQL web service endpoints, with streaming inserted into their URI paths. Therefore, YQL’s streaming-enabled endpoint for public tables is accessible through this URL:

http://query.yahooapis.com/v1/public/streaming/yql?[query_params]

whereas the streaming-enabled endpoint for OAuth-protected tables can be accessed at this URL:

http://query.yahooapis.com/v1/streaming/yql?[query_params]

The number of concurrent Comet connections has been throttled at the YQL engine: When the maximum number of concurrent Comet connections has been reached,  any requests that would normally have been put into CDP mode are served in the traditional way.

The version of the Comet implementation that CDP builds upon does not support a configurable timeout for Comet connections, with the effect that a Comet connection will remain open for only 20 seconds. This limitation will be lifted in a future YQL release.

This corruption of JSON response content would manifest itself in a number of ways: JSON numbers in the downstream response would be delivered as JSON strings to the client, and single-element JSON arrays converted to JSON objects, among others.

For example, "myint":[5] in the downstream response would be returned as "results":{"json":{"myint":"5"}} to the client.

Recent YQL releases have fixed these issues and now provide support for round trip lossless JSON processing. As we want to give developers sufficient time to take advantage of and adapt to this change, we have not yet enabled this feature by default. To enable it, simply append jsonCompat=new to your YQL query.

Please start experimenting with this new query parameter and give us feedback. Eventually, this feature will be enabled by default, but there will an announcement before we make the switch.

http://blog.pipes.yahoo.net/2011/06/10/pipes-v2-engine-timeline/