Asynchronous POST to a server

[neville]

I have an LC stack I want to bring over to OxT. At one point it uses an asynchronous POST to an LCServer, using the tsnet package installed in the pro version of LC since LCC (it is proprietary; I think it is an implementation of the tailscale Python package, or is ts someones initials??). The liburl package in LCC, so I presume also OxT, only provides synchronous access.

Hopefully at some point in the future there will be an asynchronous liburl option in OxT (can I add that as a feature request please?) but in the meantime I am going to have to use sockets. I believe I can dredge up whatever I used to know about sockets if I have to, but just in case ... has anyone on the list got any code I can crib?

[tperry2x]

I recently found liburl to be unreliable. (KBs of missing information on https transfers).
LibURL seems to be LCC's (and therefore OXT's) rebranding of cUrl, but from about 2012. (version 7.27).

As such, I use cUrl directly from a shell command. The system-installed version you might get in a package manager, via Homebrew (Mac), or via Windows update should be 8.10.1 in comparison. (Last updated Sept 18th 2024)

You could run these shell commands in a non-blocking way, making them asynchronous. You could also use curl with a POST method, with an accompanying PHP script.

[FourthWorld]

Callbacks are how LC provides async network I/O.

With libURL wrapped around tSNet:
post <data> to url <ur> with message <callbackHandlerName>

With tsNet directly:
https://lessons.livecode.com/m/4071/l/6 ... -callbacks

History:

libURL was the first solution for handling HTTP and FTP protocols using the engine's built-in socket handling.

Many years later a third party wrote tsNet, a wrapper around cURL using LC's externals interface.

Years after that tsNet was bundled with LC, and libURL was updated to use tsNet if present, and fall back to its original scripted socket routines if not.


Tip about missing bytes:
Check encoding. LC's encoding defaults differ from most others, a common source of confusion when encountering output of unexpected size.

[tperry2x]

I know of three ways. Perhaps the put binfile is old-skool, but that's the way it used to be done in earlier versions of RunRev.

All 3 methods produce different download sizes.
There should be no text encoding involved as I'm not reading in text, just reading from a URL and writing the output to a binary file. All downloading the same file.

Only the one using the curl system method is valid and opens.

methods.png (37.55 KiB) Viewed 752 times

Edit: Actually, I'll correct myself: To say that the libURL download doesn't work isn't strictly true. The LibURL download method will work 90% of the time, but doesn't seem to be 100% reliable. You'd probably want to do a checksum check of each file downloaded to verify it got it all.

Of course, this only deals with getting things from a server, not posting data to it - so a bit off topic. Sorry Neville.

[OpenXTalkPaul]

To be clear, cURL has come preinstalled with macOS for a long time now, and still does as of Sonoma macOS v14.6.1 (I just checked it )

[OpenXTalkPaul]

Callbacks are how LC provides async network I/O.

With libURL wrapped around tSNet:
post <data> to url <ur> with message <callbackHandlerName>

With tsNet directly:
https://lessons.livecode.com/m/4071/l/6 ... -callbacks

History:

libURL was the first solution for handling HTTP and FTP protocols using the engine's built-in socket handling.

Many years later a third party wrote tsNet, a wrapper around cURL using LC's externals interface.

Tip about missing bytes:
Check encoding. LC's encoding defaults differ from most others, a common source of confusion when encountering output of unexpected size.

Thanks for the Tip.
This might have been why I wound up using a 'web app' stack as my 'Resource Depot" (formerly the 'Extension Store") to transfer as base64 encoded packages, it was a reliable way I found to get to download from the GitHub repo that serves the packages content. I was having some strange issues trying to transfer the packages using libURL.

WE could do a wrapper around libCURL using xBuilder FFI, that would be a lot of work, there's a lot to it, although doing a partial wrapper that wraps only the functionality for our most common uses is an option as well.

Apple Cocoa frameworks also has NSURL which can be used for some of what one might want to do with cURL, and I already have some of that wrapped. Obviously NSURL is not a very cross-platform solution (although GNUStep does have an 'NSURL' too).

An alternative could probably be done as an XT-script-based extension library that finds and calls on the OS installed version of cURL command line tool, which in my opinion would be better because if it uses the OS installed cURL we would never need to worry about updating the binary, the OS should.

I like the idea of trying to stick with script-based Extension libraries for things like this for a couple of other reasons. Being in the packaged (.lce) format an Extension means it can easily be included when deploying a standalone that uses it. Being XT script + shell CLI based, could work just as well on LC Community engine Version 7 or 8 (for OXT 'Retro',LCC for RPi, or other non-xBuilder xTalk such as OpenXION).

The trickiest bits might be hooking it into the revLibURL library although, having looked at the source a bit, it looks like it was set up to allow for alternative back-ends at some point. So maybe it wouldn't be a hassle. I think as a 'how-to' template one might look into how they shoe-horned in revLibURL functionality for Enscripten/HTML5 (it's in a separate file, revLibURL is NOT a single script library file) which uses Javascript in browser to impliment (I guess it could be using an Enscripten / asm.js port of cURL, I'm not sure).

[neville]

Personally I've never had any problems with libURL, once I found out about the headers to prevent timeouts (which was perhaps the source of Paul's missing data?)

I'm not suggesting a version of tsnet be implemented in OxT any time soon, but eventually a libURL library which handled async access would be a Good Thing [the latest forum from across the way indicates updating their tsnet to "handle http/2" - don't know what the problem is, hasn't impacted me as far as I know - is still having problems after being promised in 2021, so it seems not to be a trivial matter].

In the meantime I guess I will be rolling my own async POST equivalent. As I recall the LC sockets thingy was not too difficult.