OpenXTalk Forums

Hi,

after some hours of fighting LC server again I had the idea that an "OpenXTalk CGI Server" might be the solution to the lack of intent to provide such, by LC Ltd.

Background:
Tests showed that LC Server is actually quite fast & useful (in CGI mode) - slower still compared to PHP 8, but not this much. Being able to use LC Server instead of PHP would be a "gift from heaven", because - well, no PHP needed ;-)

Problem:
LC Ltd., as so often. See this bug report. From 2014. Money quote from it:

Profiling with callgrind suggests that approximately 75% of LC server startup instructions are used on font loading.

This corrected, LC Server should come very close to PHP 8's performance (you really don't need any fonts when running as CGI!). But what happens? Nothing. Since 2014.

Additional problem:
Since LC Server is so crammed with dependencies (listed in this bug report) it will not run on typical shared servers. These usually are Debian or Ubuntu (some still CentOS) servers, configured as LAMP, but often stripped to the vital necessity to maximize performance & minimize attack vectors.

Such shared servers are widely used by small to middle companies - they provide email, webspace, databases, ftp and various services (ERP, CMS, blog, forum, shop, wiki ...), some even cloud services.
I don't talk of the < $3 "consumer providers" - but starting at ~$10/month there's quite some serious offers, with good performance, reliable, automated backUp, SSH & professional service. This is, for many companies, an offer too god to ignore.

But LC Server will not run on these servers, where it would be most useful. LC Ltd., when imagining a "linux server", obviously thinks of Panos' old Linux box that's mostly used to play Tux Racer.
Btw., LC Server 8 works on such "real servers" (fewer dependencies), but for whatever reasons most often still will not load "revDB.so", so has no database connection :/

Question & suggestion:
How much effort would it be to compile a "bare bone Linux-64 Server version", stripped of as many dependencies as possible, suited to run as CGI on as many servers as possible?

This would be a "unique point of selling" for the whole OpenXTalk project!
It's not that all the Jane Users & their cats would come frolicking, but there's quite some ppl that might rejoice - and quite some of these might be able to contribute here, in one way or another.
(Search terms to estimate demand would be "livecode" && "middleware", "CGI", "PHP" etc. ...)

I don't have the slightest idea of how much work this might be. But I think that a working "OpenXTalk CGI Server" would be, at this time, one of the most beneficial side-projects possible. Possibly not this much work (compared to other aspects), but maybe a relatively quick result with a huge benefit, and thus "visibility & merit" for the whole project.

I'd be ready to provide a testing ground (in one way or another, to be determined), and any help possible with testing. I could even imagine that it's possible to (moderately) charge for a "ready-to-rock" compiled version later, when it's sufficiently tested & released.

What do you think of it?

Guest wrote: Hi,

axwald , not sure why the server thought you were a guest user or why it flagged the post?
Maybe you weren't logged in? Or the log-in expired?

Anyway, yes the issues with the CGI/Server Engine has been brought up here on this forum already.

I agree, it seems quite strange that a CGI / server command line interface version of the engine loads any font handling libraries at all. I suspect this was put in as part of some attempt to please all of the people all of the time?
I guess it would be fine if this was an option, not the default. There could be a switch -F to turn ON Font handling libraries.

And I would think that compiling the CLI Server engine shouldn't be as difficult as compiling the full blown IDE with all of it's other components.

Wasn't there Linux 64 bit builds of LC Server 8? If so I'm not sure why they wouldn't be able to load the RevDB external?

To be honest, I'm probably not the right person to work on CGI aspects of this project.
I haven't made an xTalk based CGI since HyperCard + WebStar web server, sometime around 1993/1994.
But I do plan to look at this problem with CLI engine Font lib dependencies at some point.

I like @Richard Gaskin's pep talk in that bug report.

I haven't even gotten around to installing a Linux to build/test on yet.
Debian are the more strictly FOSS GPL Linux distros but it seems a lot of LC Linux users run Ubuntu based distros?

I do think Xubuntu is nice, isn't as resource intensive as some of the other desktop environments and being Ubuntu-based means DRM capabilities and other things important to a lot of regular desktop users that might not make it into Debian because of issues with licensing.

OpenXTalkPaul wrote: ↑Sun Oct 24, 2021 2:37 pm ...it seems quite strange that a CGI / server command line interface version of the engine loads any font handling libraries at all. I suspect this was put in as part of some attempt to please all of the people all of the time?

I don't think font loading was added as much as not excluded.

The replacement of the original rendering subsystem with Google's open source Skia (v7?) required deep rework, as I understand it, with a byproduct being that the CGI engine no longer omitted graphics buffering.

This has a good benefit: you can use the newer CGI engine to create custom graphics, like the once-popular e-postcard services and many others.

Very cool, but not without a downside: the graphics subsystem is also not easily turned off.

All the font file thrashing is a natural part of the graphics.

Obviously, this is a benefit to what I'd estimate is probably less than 1% of all CGI use cases.

But those who've looked into the issue in the source (Brian Wilby, Mark Wieder, and Peter Brett) report that the graphics are so intertwined with other engine parts that separating it is nontrivial.

That said, we also know that standalones can be run facelessly, by adding a -ui flag in the command line.

I don't know why that flag is available for standalones and not the CGI engine, since both work well for CGI use (though the PHP-like additions to the Server engine are attractive to some, and Server's built-in handling of things like parsing POST data is both simpler and more efficient than scripting that handing as is needed when using a standalone as a CGI).

All that said, the issue's significance has waned for me. In the years since I submitted that enhancement request I've grown tired of having to explain why I'm still using CGI to teams in other orgs I work with.

CGI is a great option for quickie things done on shared hosting, but VPSes are almost as cheap these days and allow more efficient and more scalable proxy options. And public clouds strongly favor proxies, with some services and tools (eg NgineX) providing no support for CGI at all.

If I wanted to arm wrestle with those teams I could put up a good argument for Lighttpd, and note the minor overhead of multiprocessing on Linux relative to Windows, where multiprocessing (and hence CGI) is well known to be far less efficient.

But do I want to spend valuable meeting time arm wrestling with stakeholders?

Ultimately, for orgs building out scalable systems with an eye for expansion and possible future acquisition, the conversation has proven much simpler to just adopt modern tooling.

While I do maintain some CGI-driven services (including one built with a standalone engine), new server work in 2021 and forward here is done in Node.js.

JS is rather a lovely language once you take the time to get to know it, and since it's the only language available in browsers most of us need to know it anyway. Having it available on servers, with vast repos of tools and libraries, is a nice bonus that my clients prefer given the time savings of being able to tap into an ecosystem of so much existing code.

FourthWorld wrote: ↑Sun Oct 24, 2021 8:53 pm JS is rather a lovely language once you take the time to get to know it, and since it's the only language available in browsers most of us need to know it anyway. Having it available on servers, with vast repos of tools and libraries, is a nice bonus that my clients prefer given the time savings of being able to tap into an ecosystem of so much existing code.

I've looked at a LOT of JS over the years (OK, it's been decades), and have used JS on occasion (incl. making xTalk wrappings around it) but I still don't like the dot.syntax stuff and probably never will, HOWEVER...
There are some REALLY nice libs in NPM, and not just for doing things in web browsers.
https://www.npmjs.com

So if the pre version 7 versions didn't have this issue with graphics libraries being ingrained, and you don't need to generate graphics, then why not use the v6 server engine? Because it didn't have full Unicode support? Was v6 not compiled to Linux 64bit?

And if v8 CLI Engine had acceptable performance with less dependencies then why not use that?
Was the RevDB external broken with the Server Engine in v8?

It sounds like it may be more trouble than it's worth for someone to work on that for v9.
Perhaps if someone were to work on that, it might be best to start from an earlier version and back-port the good newer features of later v9 source? I don't know what those features might be for a CLI app other than bug fixes, maybe the newer v9 Builder VM?

Hi,

OpenXTalkPaul wrote: ↑Sun Oct 24, 2021 2:37 pm axwald , not sure why the server thought you were a guest user or why it flagged the post?
Maybe you weren't logged in? Or the log-in expired?

Quite sure a "login expired". Just makes me wonder that the forum accepted the post then, at all.

OpenXTalkPaul wrote: ↑Sun Oct 24, 2021 2:37 pm And I would think that compiling the CLI Server engine shouldn't be as difficult as compiling the full blown IDE with all of it's other components.

This! It's just the engine - no IDE, nothing else. And that already would provide a desirable product that's unique.

OpenXTalkPaul wrote: ↑Sun Oct 24, 2021 2:37 pm Wasn't there Linux 64 bit builds of LC Server 8? If so I'm not sure why they wouldn't be able to load the RevDB external?

I tried quite a lot of the 8-64bit LC servers. They are the only ones that run on my shared servers at all, but none ever recognized it's revdb.so. I tried anything possible, wrote a bug report, tried even more, but any try to call any revdb-related function or command fails reliable. Money quote:

Panos wrote:I did a quick test on my Ubuntu 16.04 box, and revDB works as expected for me in LC Server 8.1.10.

So they check bug reports on a cobweb covered venerable pensioner machine :/


=== some hours later: ===

FourthWorld wrote: ↑Sun Oct 24, 2021 8:53 pm Obviously, this is a benefit to what I'd estimate is probably less than 1% of all CGI use cases.

This! And it throws LC server miles & miles back - instead of coming close to be able to compete with PHP 8 it will not even run on an out-of-the-box todays Ubuntu SERVER installation.

FourthWorld wrote: ↑Sun Oct 24, 2021 8:53 pm [...] VPS [...] public clouds [...]

Among my customers (EU) the use of VPS is nil. The costs are prohibitive - it's not the fee for the provider alone, it's the costs for proper administration, and for all the officials & certificates for data protection, data security etc. etc. that hurts.
Public cloud is out by the same bureaucratic reasons, and the "big ones" (Azure, AWS etc.) are not a choice for the many small & middle companies where a LiveCoder may find her customers.

As mentioned, a really good Shared Server comes for 10 - max. 50 €/month, this is by far enough for a web site, a web shop, some databases, the ERP, the EMail & whatever, with anything covered & included, even the often ignored daily backup.
You cannot ignore such an offer - at least until you're ready for IPO.

FourthWorld wrote: ↑Sun Oct 24, 2021 8:53 pm [...] the PHP-like additions to the Server engine [...]

Do you happen to have a link for this? I read about some "server specific merge" and such, but I'm obviously too stupid to find any link :/

OpenXTalkPaul wrote: ↑Mon Oct 25, 2021 12:51 am There are some REALLY nice libs in NPM [...]

Like this? :)

Back to topic:

OpenXTalkPaul wrote: ↑Mon Oct 25, 2021 12:51 am Was v6 not compiled to Linux 64bit?

7.0 is the first 64-bit server, and it doesn't run on any "real server" available to me. Only 8 does - but it doesn't load revdb.so, see above. 9 again doesn't run, due to dependencies overload.

I'd be more than happy with a Linux 64bit 6.7.10/11 Server! In fact, that would be the best solution at all. But are there sources anywhere?

Where a functional LC Server could shine:
Having to connect to a database isn't this uncommon a task. And it works quick & easy from LC - but the query & the results are sent in plain text. So it's of very limited use.
"revOpenDatabase" claims to be able to support SSL (param "useSSL"), but for todays mySQL/MariaDBs this doesn't work:

Panos wrote:This is because MySQL 8 supports a stronger authentication method based on
SHA256, and this method is used by default.
LiveCode (as well as some other MySQL connectors/clients) do not support
this authentication method yet.

This is from 26.8.20, on the mailing list, the time 9.6.1 came out. No idea if it's fixed yet.
So we need need a middle-ware in fact, but LC Server isn't fit for this. So we must use PHP.
Actually it's the recommended solution for LC coders to use PHP for database access. What a shame!

Whoever compiles a functional (maybe even v6 - it doesn't lack much functionality here) 64bit linux server (w/o font loading), and so releases the LC community from the shameful need to use PHP for the middleware, would achieve lasting recognition, & might be seen among icons like Sarah Reichelt or Mark Smith ;-)

Have fun!

axwald wrote: ↑Mon Oct 25, 2021 3:26 pm Hi,

OpenXTalkPaul wrote: ↑Sun Oct 24, 2021 2:37 pm axwald , not sure why the server thought you were a guest user or why it flagged the post?
Maybe you weren't logged in? Or the log-in expired?

Quite sure a "login expired". Just makes me wonder that the forum accepted the post then, at all.

Guests are allowed to post here, at least for now.

axwald wrote: ↑Mon Oct 25, 2021 3:26 pm 7.0 is the first 64-bit server, and it doesn't run on any "real server" available to me. Only 8 does - but it doesn't load revdb.so, see above. 9 again doesn't run, due to dependencies overload.

revDb external loads additional DB drivers binaries, does it not? Perhaps that is a point where a SHA256 cable DB driver could fairly easily be inserted? I'm just speculating here.

axwald wrote: ↑Mon Oct 25, 2021 3:26 pm I'd be more than happy with a Linux 64bit 6.7.10/11 Server! In fact, that would be the best solution at all. But are there sources anywhere?

Yes, I believe the source is available for every version going back to 6.0.1.
https://github.com/livecode/livecode/re ... tag/6.7.11

If I recall correctly, v.6 didn't have full unicode support, I guess that's not a problem if you don't need double byte characters or you're using ASCII friendly HTML entities.

axwald wrote: ↑Mon Oct 25, 2021 3:26 pm Where a functional LC Server could shine:
Having to connect to a database isn't this uncommon a task. And it works quick & easy from LC - but the query & the results are sent in plain text. So it's of very limited use.
"revOpenDatabase" claims to be able to support SSL (param "useSSL"), but for todays mySQL/MariaDBs this doesn't work:

Panos wrote:This is because MySQL 8 supports a stronger authentication method based on
SHA256, and this method is used by default.
LiveCode (as well as some other MySQL connectors/clients) do not support
this authentication method yet.

This is from 26.8.20, on the mailing list, the time 9.6.1 came out. No idea if it's fixed yet.
So we need need a middle-ware in fact, but LC Server isn't fit for this. So we must use PHP.
Actually it's the recommended solution for LC coders to use PHP for database access. What a shame!

Whoever compiles a functional (maybe even v6 - it doesn't lack much functionality here) 64bit linux server (w/o font loading), and so releases the LC community from the shameful need to use PHP for the middleware, would achieve lasting recognition, & might be seen among icons like Sarah Reichelt or Mark Smith

Have fun!

Perhaps some other middleware could be used. I assume the server includes Builder extension capabilities? LCC v.8 did have Builder, but it used an earlier version of its VM, and I can't recall if it had full Foreign Function Interface (which I believe is built on RedHat's libFFI) or if it was somehow limited in functionality vs. v.9.x Builder

I don't know who Sarah Reichelt is... looked her up... ah TrozWare, cool stuff!
I'm guessing Mark Smith you refer to is the guy who wrote some cool libraries (and who sadly died around 2012), and not the Mark Smith that is currently on the use-list?
Some time ago I did some work at updating, trying to complete and extend his library for reading and writing ID3 tags (commonly found in mp3s and some other media formats), available here:
https://github.com/PaulMcClernan/id3lib

I didn't start using LC until around 2013/2014, I didn't know it existed or had a community around it until around then. After HyperCard I was mostly looking at other things, AppleScript/ASObjC, and at one point purchased a SuperCard license (which I still have an SC v.4.7.3 install that works up to macOS 10.14.6)

axwald wrote: ↑Mon Oct 25, 2021 3:26 pm

OpenXTalkPaul wrote: ↑Mon Oct 25, 2021 12:51 am There are some REALLY nice libs in NPM [...]

Like this?

Back to topic:

Could, and has, happen with a lot of languages ( including xTalk(s) as there was a security flaw in LC for Win from Oct. 2020, looks like patched in July 2021) and other package managers, even Apple's App Store.
https://nvd.nist.gov/vuln/detail/CVE-2020-26894

The nice thing about things that are open source is that there's a good chance users or security researchers will catch something like that and it gets patched quickly.

But I was thinking more of things that have nothing to do with building web sites, like this:
https://www.npmjs.com/package/rtpmidi

Hey guys
interesting discussion, thanks.

I also learn from it that people are interested in totally different features of the platform than what I would focus on,
also some that I had seen as strategically minor because mainstream platforms do it better - like why grieve over encryption of database connections and non-CGI server components when the big ones are so much ahead? Of course such features are a total must-have in the longer run, but would somebody care if OXT can do it too next week and then switch away from a mainstream professional developer platform backed by big names? Unlikely, I would say.

I would focus on domains where LC/OXT shines, namely rapid cross-platform GUI-based work. I still have not found another platform that can match what I see here, and you bet I looked around.

Of course there would be nothing to say against closing gaps an OXT code base has. Just given very limited resources in this place the question is what is the most urgent need? How would it help the long-term success and survival of the platform if we just _only_ have a compilation of a new 64-bit server edition running on current Ubuntu, as nice to have this surely would be?

If anything shall happen at all this OXT effort will need strategy and structure and a step-by-step mentality eventually.
A single successful compilation of the code base as-is by a non-LC party is key. This would open the door, but it would also not suffice. Perhaps it has happened already behind the scenes somewhere. Some of the usual suspects and people that could help doing it have been named in this thread. There sure are more, but they would have to be involved first.

A first compilation step, as big as it is, would not suffice as all people including ourselves that are interested in deploying OXT stuff on a regular basis could not later rely on one individual who would _maybe_ do a new compilation of some part of something eventually in his random spare-time but they would want to be sure that changes in the code result in a new downloadable binary in a certain timeframe. Such results could only be expected from documented compilation know-how and infrastructure taken care of by a group of people with a common interest in OXT. The forum to organize this would be here. If we do not do it, it will maybe not happen.

If all the effort does not go this way OXT will only be a play thing (which is ok!), it will not be a notable open-source project, and as APIs and hardware platforms grow away from the LC 9.6.3 status, the less interesting the code will become for professional work or even for educational purposes (where gaps in features might not be critical) over time. This is ok, we can buy a LC license (as long as it will be offered) with exactly the features LC Ltd decided to focus on and managed to produce. Hm, well.

As you ask for clearly professional features (like fast secured server connections) I would assume you do it because you have potential professional work in mind. So either there is professional discussion of near-term important goals and how to reach them best, or even the long-term very important features will likely never be added because there is no code base and compilation machine for on-going development.

Hope the de-branding efforts are going ahead well. Keep up all the good work!

mdm wrote: ↑Mon Oct 25, 2021 11:04 pm Hey guys
interesting discussion, thanks.

I also learn from it that people are interested in totally different features of the platform than what I would focus on,
also some that I had seen as strategically minor because mainstream platforms do it better - like why grieve over encryption of database connections and non-CGI server components when the big ones are so much ahead? Of course such features are a total must-have in the longer run, but would somebody care if OXT can do it too next week and then switch away from a mainstream professional developer platform backed by big names? Unlikely, I would say.

It would be nice if xTalk could suddenly supplant something like PHP as a widely used tool, but I agree, not likely to happen any time soon.

mdm wrote: ↑Mon Oct 25, 2021 11:04 pm I would focus on domains where LC/OXT shines, namely rapid cross-platform GUI-based work. I still have not found another platform that can match what I see here, and you bet I looked around.
---SNIP---
Hope the de-branding efforts are going ahead well. Keep up all the good work!

I also agree that where it shines most, and always has, is as an instant-compiling Rapid Application Dev Environment with live GUI editing. Although I also understand the appeal of using the chunking text handling language for doing stuff purely with text, seems perfect for web/CGI usage.

It does seem still very much true that xTalk has a diverse user base with variety interests.
As Bill Atkinson once said it's a "Solution looking for a problem"!

I would like nothing more than to not worry about compiling the engines, finish the de-branding (or rebranding where de-branding is not possible) which again must be the first step, and then work on improving the IDE and docs (which has been part of that first step), and move on to expanding the possibilities available with more widgets and wrapping a plethora of foreign libraries in Builder.

I've been busy with the "day job" and other things for the past last week or so, but I have made a lot of progress with the IDE, and yesterday someone sent me some files which would make a good "Stack Ideas" stack that could be included with OXT, as well as pointing out a few areas where more decoupling from LC Ltd is needed. Made me happy just to have someone new interested and donating work.

axwald wrote: ↑Mon Oct 25, 2021 3:26 pm Among my customers (EU) the use of VPS is nil. The costs are prohibitive - it's not the fee for the provider alone, it's the costs for proper administration, and for all the officials & certificates for data protection, data security etc. etc. that hurts.
Public cloud is out by the same bureaucratic reasons, and the "big ones" (Azure, AWS etc.) are not a choice for the many small & middle companies where a LiveCoder may find her customers.

I respect that server admin isn't everyone's cup of tea, and why SAAS solutions are a growing opportunity. But while I can appreciate saving time, I would caution against any system connected to the open Internet with customer data managed by a developer unfamiliar with sys admin. Security and privacy are ongoing processes that affect all development decisions.

This is one of the challenges of xTalks in the Internet Age: they were invented back when software meant one user on one machine. Security wasn't a considerations; MacOS didn't even have a login screen in those days. We just can't code with that mindset any more. Even for single-user apps, we have to consider laptop theft, other processes that may have crept into the system, and other factors that can affect user security and privacy. With networked apps the considerations multiply. Tedious, yes, but just the price we play to enjoy the modern luxury of having the entire world connected by a shared network. Younger devs pick these things up quickly, having been immersed in this connected world from their first day.

As mentioned, a really good Shared Server comes for 10 - max. 50 €/month, this is by far enough for a web site, a web shop, some databases, the ERP, the EMail & whatever, with anything covered & included, even the often ignored daily backup.
You cannot ignore such an offer - at least until you're ready for IPO.

IPO is an uncommon exit strategy; acquisition more likely, what most aim for. Shared hosting is great for small sites which have no ambitions for scaling, but the exclusion of any persistent processes plus the "noisy neighbor" problem characteristic of shared hosting are among the reasons that's rarely used for acquirable services.

In all fairness, it's not always a show-stopper. I have had one client reach successful acquisition with a product delivered on a shared host using LC CGIs. But the acquiring company has complained about it ever since, and we are in the process of moving to reverse proxy running a different scripting engine.

FourthWorld wrote: ↑Sun Oct 24, 2021 8:53 pm [...] the PHP-like additions to the Server engine [...]

Do you happen to have a link for this? I read about some "server specific merge" and such, but I'm obviously too stupid to find any link :/

I had not considered this question before. If you've used any other language engine as a CGI -- even MetaCard or LC standalones, along with Python, Perl, etc. -- I would have thought the similarities self-evident.

So I'm not sure where to begin.

The implicit merge (putting code and content into a single page) is perhaps the clearest example, but the engine's processing of incoming GET args and POST data are also similar, as is the practice of mirroring environment vars in an internal array.

Let me know if that doesn't answer the question and I may be able to expand on that.

FourthWorld wrote: ↑Wed Oct 27, 2021 6:49 pm We just can't code with that mindset any more. Even for single-user apps, we have to consider laptop theft, other processes that may have crept into the system, and other factors that can affect user security and privacy.

I think that depends on what you're coding. My coding has mostly been about fun and games, playing with Input devices (Joysticks and Musical instruments), making knobs, drum machine / piano roll style grids, and a few media oriented utilities, GUIs for CLIs, etc. ... no bank accounts involved, rarely even local networks used.

I don't envy you people dealing with that stuff.

OpenXTalkPaul wrote: ↑Wed Oct 27, 2021 8:12 pm

FourthWorld wrote: ↑Wed Oct 27, 2021 6:49 pm We just can't code with that mindset any more. Even for single-user apps, we have to consider laptop theft, other processes that may have crept into the system, and other factors that can affect user security and privacy.

I think that depends on what you're coding. My coding has mostly been about fun and games, playing with Input devices (Joysticks and Musical instruments), making knobs, drum machine / piano roll style grids, and a few media oriented utilities, GUIs for CLIs, etc. ... no bank accounts involved, rarely even local networks used.

I don't envy you people dealing with that stuff.

It's pretty great, actually. But like anything else, it depends on your personal interests.

I love seeing what happens when we unleash a tool inside a company and the collaboration starts flowing.

But yes, the moment the feature set includes a login, you've entered a world that taps into your desire to learn new things and to protect others. If those are your instincts, the connected world is a wonderful playground of opportunity.