1. Changes
   1. More strict processing of geometry changes, repaints and recording
   2. Dynamic slots for medium state and the current text line
   3. Thread-safe output history and drawing context
   4. Refactored text renderer for performance and alternative text directions
   5. New demos
2. Future changes
   1. The repaint queue
   2. Input editing streams rewrite
   3. SDL2 backend and keyboard layouts
3. Closing remarks

Dear All,

After a year of development it is time to make a release. The codename of this version is "Ostara", after the second of the eight annual pagan holidays celebrating the arrival of spring. Ostara was a Germanic goddess representing rebirth and is the equivalent of Easter.

In this release we mainly focus on robustness with numerous bugs fixed and a few features implemented. Most notably McCLIM streams are now thread-safe to write to and to draw on. Here is the list of committers and their number of commits since the last release:

Daniel Kochmański (502), Charlie McMackin (15), Andrea Demichele (2), John Carroll (1), José M. Á. Ronquillo Rivera (1) and Robert Brown (1).

This is not an exhaustive list of contributors, as it does not list people who reported issues, updated the wiki, or performed other non-coding tasks. Thank you! Moreover I'd like to thank people who donate money to my Patreon page, your continuous support means a lot to me.

Changes

For a detailed list of changes, consult the NEWS file or the git log directly. Below is a high-level overview of these changes.

More strict processing of geometry changes, repaints and recording

This is the least visible change because it pertains to how we process updates to states of CLIM stream panes. Previously when the program added new output, the pane was resized immediately, followed by scrolling and repainting. That was clearly suboptimal, especially when there is a lot of changes done in a short period of time. Instead, we now batch these changes and execute them when STREAM-FINISH-OUTPUT is called. That greatly improves performance and saves visual glitches for certain scenarios.

Dynamic slots for medium state and the current text line

To enable concurrent drawing threads, we needed to ensure that there are no races regarding drawing options (like the drawing ink or the text style). To do that, CLIM stream panes have a slot that contains an instance of a class that has dynamic slots (for how this work see my blog post). This allows for drawing and writing text to streams concurrently from multiple threads. For example:

In this demo we have seven concurrent threads:

• two threads print 1000 lines with prefixes "XXX" and "YYY"
• four threads drawing dots in different colors
• one thread monitors the current progress of printing lines

Note that we may also scroll the output while these threads are still running and modifying the sheet. The cursor is not corrupted and the output proceeds without glitches.

Thread-safe output history and drawing context

Modifications to the output record history and writing pixels on a buffer are mutually exclusive operations protected with locks. For forward compatibility, all repaint requests by the program should be issued with DISPATCH-REPAINT. Moreover the user program should not directly use operators that change the output recording state of the stream.

Notice how multiple threads are modifying the sheet geometry and the cursor position. The output record history is a bottleneck because each thread, after drawing, needs to take the lock and update the record. This is suboptimal, but enables us to quickly write throwaway code like this:

(in-package "CLIM-USER")

(defparameter *stream* (open-window-stream :width 800 :height 600))
(defparameter *stop* t)

(defun woosh (id ink)
  (setf *stop* nil)
  (with-drawing-options (*stream* :ink ink)
    (loop while (null *stop*) do
      (updating-output (*stream* :unique-id id :fixed-position t)
        (format *stream* "Time: ~a~%" (get-internal-real-time)))
      (sleep 1/30))))

(bt:make-thread (lambda () (woosh :thread-1 +dark-red+)))
(bt:make-thread (lambda () (woosh :thread-2 +dark-blue+)))

and in my opinion it is great for prototyping, because you can add additional elements while the program is running without invoking arcane protocols.

Refactored text renderer for performance and alternative text directions

The text renderer has been refactored to achieve better performance and exploit optimization opportunities when they are presented. For example a different code path is executed when the text is not translucent.

That said, the most notable change with the text renderer is the ability to change the line and the page directions:

These changes apply to DRAW-TEXT. Line-oriented integration with CLIM streams that allows mixing the text that has different directions is not yet implemented, but we can already mix text and graphics, and the latter will be correctly wrapped along the text when necessary.

New demos

I wrote new demos that may be started from the system clim-examples:

• incremental redisplay: demonstrates how to work with output record caches
• animation with pulse: an animation with a timer synchronizing redisplay
• concurrent drawing: a demo where we can spin any number of drawing threads
• concurrent writing: a stress test for concurrent writing lines to a stream
• concurrent griding: a stress test with concurrent drawing with many records

Future changes

The repaint queue

The thread-safety of CLIM streams is factored out from a branch repaint-queue. This branch works, but it still has various regressions that need to be addressed before merging. The results are very promising and its effect compounds with thread-safety, because drawing locks the output history far less often, while recording is rather cheap.

Input editing streams rewrite

This work is mostly done but it needs more testing and there are a few bugs that need to be addressed before merging. This change involves also better command completions that include a fuzzy completer, and makes text editing much easier. If you want to see the current state of it check out input-editing branch.

SDL2 backend and keyboard layouts

I didn't work on either of these tasks since the last release, but I plan to finish them after merging branches repaint-queue and input-editing. After finishing SDL2 branch and keyboard layouts I want to make the next release.

Closing remarks

McCLIM improves on a daily basis, and this trend continues. I'm very happy that I've been able to make a timely release unlike the last one. Fixing numerous issues further accelerates other improvements. From my subjective perspective the software is much more resilient than a year ago.

You may grab the latest release from here: https://codeberg.org/McCLIM/McCLIM/releases/tag/0.9.9-ostara.

Best regards,
Daniel Kochmański

After over 5 years of development we've decided to make a release. The last release version "0.9.7 Imbolc" dates to 2018. Yule is a Germanic Midwinter festival that was later incorporated into Christmas.

This release is long overdue. The reason for that is because we've wanted to have too many things finished in it - I'll elaborate on these later. This release would not happen without contributions from many people who wrote code, reported bugs and provided guidance to beginners.

Changes

Here is the list of commiters and the number of commits since the last release:

Daniel Kochmański (1667), Jan Moringen (582), Elias Mårtenson (182), Nisar Ahmad (90), Andrea Demichele (89), Cyrus Harmon (62), José Ronquillo Rivera (18), Christoph Keßler (18), Eric Timmons (16), death (12), Robert Strandh (11), Phil Mueller (7), Henry Harrington (5), hes (5), chuchana (4), Karsten Poeck (4), Jingtao Xu (3), Pouar (3), bmansurov (3), contrapunctus-1 (3), Charles Zhang (2), Fred Gilham (2), contrapunctus (2), duuqnd (2), goose121 (2), Howard Shrobe (2), Alan Zimmerman (1), Dirk Eßer (1), Ingo Krabbe (1), Javier Olaechea (1), Jeremiah LaRocco (1), John Lorentzson (2), Knut Olav Bøhmer (1), Kyle Nusbaum (1), Mark Evenson (1), Nathan Shostek (1), Philipp Marek (1), Tarn W. Burton (1), William S. Annis (1), crunsk (1), luyiranl2010 (1), modula t. worm (1), ormf (1), rainthree (1), steiner (1), Неточка Незванова (1).

A few opinionated feature highlights that may be interesting to users:

• The new inspector Clouseau written by Jan Moringen
• Inclusion of the Freetype renderer by Elias Mårtenson
• New extension "mcclim-dot" by Eric Timmons (graphviz layout for graphs)
• Rewrite of the X11 renderer and a new SVG backend by Daniel Kochmański
• Many important UX improvements by Nisar Ahmad and Andrea Demichele

Here is a more complete (yet still incomplete and more noisy) list of changes. For more details please consult the repository history log.

• Listener UX improvements
• Refactor command tables and commands parsing
• Refactor of the sheet and the mirror implementation
• Numerous new examples
• Unification of CLX backends into a single system
• bugfix: TTF font caches do not overlap between backends
• Add an alternative (not default) TTF renderer based on FFI to FreeType
• bugfix: address numerous issues in tab-layout
• Improved support for text transformations
• Bundle cl-dejavu fonts as a dependency
• Portability fixes for CL implementations
• Better separation of responsibilities between sheets and mediums
• TABLE-FORMATTING improvements
• Support for system clipboard and for internal selection
• Remove invalid assumptions about the sheet coordinates while scrolling
• Full rewrite of CLX renderer to use XRender (transparency, transformations, clipping)
• Incremental redisplay partial rewrite and bug fixes
• UX imrovements for gadget states with regard to pointer events
• Bug fixes and conformity improvements for PS and PDF backends
• Pull bezier curve and bezier area extensions as core regions
• Improved set of colors for MAKE-CONTRASTING-INKS
• Improved set of patterns for MAKE-CONTRASTING-DASH-PATTERNS
• A new protocol for handling patterns, transformations and arbitrary designs
• Rewrite mcclim-render extension
• Enable rendering directly to the pattern or a raster file
• Flesh out the protocol to implement indirect inks
• Remove obsolete backend "beagle"
• Rewrite of the demo DRAWING-TESTS
• Improved support for arcs, angles and ellipses
• Geometry module partial rewrite - conformance and performance improvements
• Improved class hierarchy for input events
• Space requirements rewrite to allow specification of padding and margins
• Menu bar rewrite to handle keyboard accelerators and arrows
• Remove runtime overhead from protocol classes
• More conforming handling of line style unit
• Introduce the concept of a page with four margins
• Implement word wrap and extend FILLING-OUTPUT and INDENTING-OUTPUT macros
• Rewrite PRESENTATION-SUBTYPEP and PRESENTATION-TYPEP for conformance
• Much faster TTF renderer, implement kerning and tracking, thread safe rendering
• Pixel-perfect FONT-TEXT-EXTENT implementation for the TTF renderer, transformations
• Gadgets UX improvements for both space composition and event handling
• Better PRINT-OBJECT methods for numerous internal objects
• Rewrite EVENT-QUEUE for thread safety and performance, extend TIMEOUT interface
• Fix McCLIM regression to work on non-SMP systems
• Improved parsing for space requirements
• Rewrite the implementation of mapping between text styles and fonts
• Refactor of the EXTENDED-INPUT-STREAM
• Add numerous missing ACCEPT-PRESENT-DEFAULT methods for dialog-views
• Improved repaint-sheet handling (less repaints, more performant)
• Improvements to numerous presentation methods in functions ACCEPT and PRESENT
• Presentation translator lookup rewrite
• Selection API can carry arbitrary presentations with "paste" translators
• Extensions documentation in manual, various fixups and improvements
• Specification clarifications in the bundled latex source code
• Implement double buffering in the CLX backend
• New protocols to set the frame (and sheet) pretty name and icon
• Refactor mirrored/non-mirrored event distribution and add synthetic events
• Rewrite pointer tracking, pointer grabbing and drag&drop support
• Include the new inspector "Clouseau" with bundled applications, remove old the inspector
• Put Franz-specific extensions to the system "mcclim-franz"
• A new command parameter OUTPUT-DESTINATION to redirect the command output
• Improve frame and panes definitions to allow dynamic reinitialization
• Improve PROCESS-NEXT-EVENT specification and implementations for clarity and completness
• It is possible to define a presentation to command translator with the argument
• Rework accepting values to remove a few kludges
• Refactor FORMAT-GRAPH-FROM-ROOT and the layout algorithm for correctness
• Numerous cleanups to avoid accessing internal symbols and have a clean file structure
• Introduce the presentation BLANK-AREA (similar to NULL-PRESENTATION) that allows parameters
• Provide an uniform interface for mapping over command table menu items and keystrokes
• Improve the protocol for defining new ports (backends)
• Allow for existence of ungrafted mediums
• Allow runtime replacement of the menu bar in the frame
• Better separation of responsibilities between frames and frame managers
• Improved gesture and event matching that allows wildcards
• Minor integrations between CLIM-DEBUGGER and Clouseau
• Removal of numerous internal obsolete interfaces
• Thread-safe implementation of the function EXECUTE-FRAME-COMMAND
• Better support for repeated grafting and degrafting sheets (migrating frames between backends)
• Implement a new backend that creates SVG documents
• Introduce a new macro for drawing backends CLIME:WITH-OUTPUT-TO-DRAWING-STREAM
• Include a new extension "mcclim-tooltips"
• Include a new extension "mcclim-dot"
• Write new implementation of text-field and text-editor gadgets (with pointer selection etc)
• Allow for preloaded fonts in the application image
• Flesh out important protocols useful for writing new backends
• Add a new gesture types :TIMER, :INDIRECT, :POINTER-MOTION and :POINTER-SCROLL
• TTF renderer consults the graft DPI to decide on the font scaling

Work-in-progress features

I've mentioned some features that I've wanted to include in this release - that delayed it by over two years. I'll elaborate on them now, because they are all in a quite advanced "work in progress" state:

SDL2 backend

The SDL2 backend is an alternative backend that is meant to work on platforms that do not feature X11. It is mostly complete, but since SDL2 text input has a bad protocol, we need to rely only on keyboard events. To handle text correctly we need to transform them to text.

The current version of the backend features the software renderer. Hardware acceleration will come after that. In parallel I'm also writing a manual for developers that explains how to write new backends.

Keyboard layouts

CLX does not implement the XKB protocol. That means that McCLIM applications don't benefit from keyboard layouts. To address that I'm writing a library that parses xkb definitions and implements layouts like the project xkbcommon does. Such library will also enable the SDL2 backend (see above).

Input editing streams rewrite

McCLIM implements interactive stream handling with help of Drei. I've already written the new input editor that is better integrated with the system and it is used to implement gadgets. Integrating this editor with streams will enable improvements to the command line processor and accepting values streams.

Separate repaint queue

I've already made a few prototypes that implement animations using a separate repaint queue. They show a nice performance of 400 FPS. The biggest win will be that the input processing and the sheet repaint will not block each other.

Closing remarks

McCLIM improves on daily basis and we hope that this trend will continue. I'm sorry for dragging the release - I'll try to make releases more often so that changes may be fleshed out in more details and on more timely basis. I'd like to thank all contributors and people who supported us along the way.

I'd like to wish you all a Happy New Year :-)

download

Best regards,
Daniel Kochmański

Our repository is migrated to a new hosting platform Codeberg: https://codeberg.org/McCLIM/McCLIM. The old repository is archived in read-only state - that means that old links and references will work.

I've wanted to migrate McCLIM for quite some time and planned that for after a release, but it is frequently postponed by one-more-thing to tweak, so I've decided to take the action now.

As for the reason - we've discussed it a few times on IRC. It boils down to the fact that Github/Microsoft uses our code without attribution and without respecting its license, in order to improve its product "copilot".

Thank you for your understanding, sorry for the inconvenience(!), and see you in the new place. :-)

Sincerely yours,
Daniel Kochmański

A lot of time passed since the last blog entry. I'm sorry for neglecting this. In this post, I'll try to summarize the past two and a half year.

Finances and bounties

Some of you might have noticed that the bounty program has been suspended. The BountySource platform lost our trust around a year ago when they have changed their ToS to include:

If no Solution is accepted within two years after a Bounty is posted, then the Bounty will be withdrawn, and the amount posted for the Bounty will be retained by Bountysource.

They've quickly retracted from that change, but the trust was already lost. Soon after, I've suspended the account and all donations with this platform were suspended. BountySource refunded all our pending bounties.

All paid bounties were summarized in previous posts. Between 2016-08-16 and 2020-06-16 (46 months of donations) we have collected in total $18700. The Bounty Source comission was 10% collected upon withdrawal - all amounts mentioned below are presented for before the comission was claimed.

During that time $3200 was paid to bounty hunters who solved various issues in McCLIM. The bounty program was a limited success - solutions that were contributed were important, however harder problems with bounties were not solved. That said, a few developers who contribute today to McCLIM joined in the meantime and that might be partially thanks to the bounty program.

When the fundraiser was announced, I've declared I would withdraw $600 monthly from the project account. In the meantime I've had a profitable contract and for two years I stopped withdrawing money. During the remaining three years I've withdrawn $15500 ($440/month) from the account.

As of now we don't have funds and there is no official way to donate money to the project (however, this may change in the near future). I hope that this summary is sufficient regarding the fundraiser. If you have further questions, please don't hesitate to contact me, and I'll do my best to answer them.

I'd like to thank all people who donated to the project. Your financial support made it possible for me to work on the project more than I would be able without it. The fact that people care about McCLIM enough to contribute to it money gives me the motivation and faith that working on the codebase is an important effort that benefits the Common Lisp community.

Improvements

The last update was in 2018-12-31. A lot of changes accumulated in the meantime.

• Bordered output bug fixes and improvements -- Daniel Kochmański
• Gadget UX improvements (many of them) -- Jan Moringen
• Text styles fixes and refactor -- Daniel Kochmański
• Freetype text renderer improvements -- Elias Mårtenson
• Extended input stream abstraction rewrite -- Daniel Kochmański
• Implementation of presentation methods for dialog-views -- admich
• Encapsulating stream missing methods implementation -- admich
• indenting-output-stream fixes -- Jan Moringen
• drawing-tests demo rewrite -- José Ronquillo Rivera
• Line wrap on the word boundaries -- Daniel Kochmański
• New margin implementation (extended text formatting) -- Daniel Kochmański
• Presentation types and presentation translators refactor -- Daniel Kochmański
• Input completion and accept methods bug fixes and reports -- Howard Shrobe
• Clipboard implementation (and the selection translators) -- Daniel Kochmański
• CLIM-Fig demo improvements and bug fixes -- Christoph Keßler
• The pointer implementation (fix the specification conformance) -- admich
• Drei kill ring improvements -- Christoph Keßler
• McCLIM manual improvements -- Jan Moringen
• Frame icon and pretty name change extensions -- Jan Moringen
• Cleanups and extensive testing -- Nisar Ahmad
• pointer-tracking rewrite -- Daniel Kochmański
• drag-and-drop translators rewrite -- Daniel Kochmański
• Complete rewrite of the inspector Clouseau -- Jan Moringen
• Rewrite of the function distribute-event -- Daniel Kochmański and Jan Moringen
• Adding new tests and organizing them in modules -- Jan Moringen
• Various fixes to the delayed repaint mechanism -- Jan Moringen
• CLX backend performance and stability fixes -- Christoph Keßler
• PS/PDF/Raster backends cleanups and improvements -- Jan Moringen
• Drei regression fixes and stability improvements -- Nisar Ahmad
• Geometry module refactor and improvements -- Daniel Kochmański
• Separating McCLIM code into multiple modules -- Daniel Kochmański and Jan Moringen
• Frames and frame managers improvements -- Jan Moringen and Daniel Kochmański
• Frame reinitialization -- Jan Moringen
• PDF/PS backends functionality improvements -- admich
• Menu code cleanup -- Jan Moringen
• Pane geometry and graph formatting fixes -- Nisar Ahmad
• Numerous CLX cleanups and bug fixes -- Daniel Kochmański and Jan Moringen
• Render backend stability, performance and functionality fixes -- Jan Moringen
• Presentation types more strict interpretation -- Daniel Kochmański
• External Continuous Integration support -- Jan Moringen
• Continuous Integration support -- Nisar Ahmad
• Improved macros for recording and table formatting -- Jan Moringen
• Better option parsing for define-application-frame -- Jan Moringen
• Separation between the event queue and the stream input buffer -- Daniel Kochmański
• Examples cleanup -- Jan Moringen
• Graph formatting cleanup -- Daniel Kochmański
• Stream panes defined in define-application-frames refactor -- admich
• Menu bar rewrite (keyboard navigation, click to activate) -- Daniel Kochmański
• Thread-safe execute-frame-command function -- Daniel Kochmański
• Mirroring code simplification for clx-derived backends -- Daniel Kochmański
• Arbitrary native transformations for sheets (i.e. zooming) -- Daniel Kochmański
• extended-streams event matching improvements -- Jan Moringen
• Render backend performance improvements -- death
• drei fixes for various issues -- death
• drei various cleanups -- Jan Moringen
• clim-debugger improvements -- Jan Moringen
• Manual spelling fixes and proofreading -- contrapunctus

This is not an exhaustive list of changes. For more details, please consult the repository history. Many changes I've introduced during this iteration were a subject of a careful (and time-consuming) peer review from Jan Moringen which resulted in a better code quality. Continuous integration provided by Nisar Ahmad definitely made my life simpler. I'd like to thank all contributors for their time and energy spent on improving McCLIM.

Pending work

If you are working on some exciting improvement for McCLIM which is not ready, you may make a "draft" pull request in the McCLIM repository. Currently, there are three such branches:

• the SLIME-based backend for CLIM by Luke Gorrie

• the dot-based graph layout extension by Eric Timmons

• the xrender backend by Daniel Kochmański

Other than that, I've recently implemented the polygon triangulation algorithm that is meant to be used in the xrender backend (but could be reused i.e. for opengl). Currently, I'm refining the new rendering for clx (xrender). After that, I want to introduce a portable double buffering and a new repaint queue. Having these things in place after extensive testing, I want to roll out a new release of McCLIM.

Sincerely yours,
Daniel Kochmański

Winter solstice is a special time of year when we gather together with people dear to us. In pagan tradition this event is called "Yule". I thought it is a good time to write a progress report and a summary of changes made since the last release. I apologise for infrequent updates. On the other hand we are busy with improving McCLIM and many important (and exciting!) improvements have been made in the meantime. I'd love to declare it a new release with a code name "Yule" but we still have some regressions to fix and pending improvements to apply. We hope though that the release 0.9.8 will happen soon.

We are very excited that we have managed to resurrect interest in McCLIM among Common Lisp developers and it is thanks to the help of you all - every contributor to the project. Some of you provide important improvement suggestions, issue reports and regression tests on our tracker. Others develop applications with McCLIM and that helps us to identify parts which need improving. By creating pull requests you go out of your comfort zone to help improve the project and by doing a peer review you prevent serious regressions and make code better than it would be without that. Perpetual testing and frequent discussions on #clim @ irc.libera.chat help to keep the project in shape. Financial supporters allow us to create bounties and attract by that new contributors.

Finances and bounties

Speaking of finances: our fundraiser receives a steady stream of funds of approximately $300/month. We are grateful for that. Right now all money is destined for bounties. A few times bounty was not claimed by bounty hunters who solved the issue – in that case I've collected them and re-added to funds after talking with said people. Currently our funds available for future activities are $3,785 and active bounties on issues waiting to be solved are $2,850 (split between 7 issues). We've already paid $2,450 total for solved issues.

Active bounties:

• [$600] drawing-tests: improve and refactor (new!).
• [$600] streams: make SEOS access thread-safe (new!).
• [$500] Windows Backend.
• [$450] clx: input: english layout.
• [$300] listener: repl blocks when long process runs.
• [$150] UPDATING-OUTPUT not usable with custom gadgets.
• [$150] When flowing text in a FORMATTING-TABLE, the pane size is used instead of the column size.

Claimed bounties (since last time):

• [$100] No applicable method for REGION-CONTAINS-POSITION-P -- fixed by Cyrus Harmon and re-added to the pool.
• [$200] Text rotation is not supported -- fixed by Daniel Kochmański.
• [$400] Fix Beagle backend -- cancelled and re-added to the pool.
• [$100] with-room-for-graphics does not obey height for graphics not starting at 0,0 -- fixed by Nisar Ahmad.
• [$100] Enter doesn't cause input acceptance in the Listener if you hit Alt first -- fixed by Charles Zhang.
• [$100] Listener commands with "list" arguments, such as Run, cannot be executed from command history -- fixed by Nisar Ahmad.
• [$200] add PDF file generation (PDF backend) -- fixed by Cyrus Harmon; This bounty will be re-added to the pool when the other backer Ingo Marks accepts the solution.

Improvements

I'm sure you've been waiting for this part the most. Current mid-release improvements and regressions are vast. I'll list only changes which I find the most highlight-able but there are more and most of them are very useful! The whole list of commits and contributors may be found in the git log. There were also many changes not listed here related to the CLX library.

• Listener UX improvements by Nisar Ahmad.
• Mirrored sheet implementation refactor by Daniel Kochmański.
• New demo applications and improvements to existing ones,
• Font rendering refactor and new features:

This part is a joint effort of many people. In effect we have now two quite performant and good looking font rendered. Elias Mårtenson resurrected FFI Freetype alternative text renderer which uses Harfbuzz and fontconfig found in the foreign world. Daniel Kochmański inspired by Freetype features implemented kerning, tracking, multi-line rendering and arbitrary text transformations for the native TTF renderer. That resulted in a major refactor of font rendering abstraction. Missing features in the TTF renderer are font shaping and bidirectional text.

• Experiments with xrender scrolling and transformations by Elias Mårtenson,
• Image and pattern rendering refactor and improvements by Daniel Kochmański.

Both experiments with xrender and pattern rendering were direct inspiration for work-in-progress migration to use xrender as default rendering mechanism.

Patterns have now much better support coverage than they used to have. We may treat pattern as any other design. Moreover it is possible to transform patterns in arbitrary ways (and use other patterns as inks inside parent ones). This has been done at expense of a performance regression which we plan to address before the release.

• CLX-fb refactor by Daniel Kochmański:

Most of the work was related to simplifying macrology and class hierarchy. This caused small performance regression in this backend (however it may be fixed with the current abstraction present there).

• Performance and clean code fixes by Jan Moringen:

Jan wrote a very useful tool called clim-flamegraph (it works right now only on SBCL). It helped us to recognize many performance bottlenecks which would be hard to spot otherwise. His contributions to the code base were small (LOC-wise) and hard to pin-point to a specific feature but very important from the maintanance, correctness and performance point of view.

• Text-size example for responsiveness and UX by Jan Moringen,
• Various gadget improvements by Jan Moringen,
• Box adjuster gadget rewrite by Jan Moringen:

clim-extensions:box-adjuster-gadget deserves a separate mention due to its usefulness and relatively small mind share. It allows resizing adjacent panes by dragging a boundary between them.

• New example for output recording with custom record types by Robert Strandh,
• PostScript and PDF renderer improvements by Cyrus Harmon,
• Scrigraph and other examples improvements by Cyrus Harmon,
• Multiple regression tests added to drawing-tests by Cyrus Harmon,
• Ellipse drawing testing and fixes by Cyrus Harmon,
• Better contrasting inks support by Jan Moringen,
• Output recording and graphics-state cleanup by Daniel Kochmański,
• WITH-OUTPUT-TO-RASTER-IMAGE-FILE macro fixed by Jan Moringen,
• Regions may be printed readably (with #. hack) by Cyrus Harmon,
• event-queue processing rewrite by Nisar Ahmad and Daniel Kochmański:

This solves a long standing regression – McCLIM didn't run correctly on implementations without support for threading. This rewrite cleaned up a few input processing abstractions and provided thread-safe code. SCHEDULE-EVENT (which was bitrotten) works as expected now.

• Extensive testing and peer reviews by Nisar Ahmad:

This role is easy to omit when one looks at commits but it is hard to overemphasize it – that's how important testing is. Code would be much worse if Nisar didn't put as much effort on it as he did.

Plans

Before the next release we want to refactor input processing in McCLIM and make all stream operations thread-safe. Refactoring input processing loop will allow better support for native McCLIM gadgets and streams (right now they do not work well together) and make the model much more intuitive for new developers. We hope to get rid of various kludges thanks to that as well. Thread-safe stream operations on the other hand are important if we want to access CLIM application from REPL in other process than the application frame (right now drawing from another process may for instance cause output recording corruption). This is important for interactive development from Emacs. When both tasks are finished we are going to release the 0.9.8 version.

After that our efforts will focus on improving X11 backend support. Most notably we want to increase use of the xrender extension of clx and address a long standing issue with non-english layouts. When both tasks are accomplished (some other features may land in but these two will be my main focus) we will release 0.9.9 version.

That will mark a special time in McCLIM development. Next release will be 1.0.0 what is a significant number. The idea is to devote this time explicitly for testing, cleanup and documentation with a feature freeze (i.e no new functionality will be added). What comes after that nobody knows. Animations? New backends? Interactive documentation? If you have some specific vision in which direction McCLIM should move all you have to do is to take action and implement the necessary changes :-).

Merry Yule and Happy New Year 2019

This year was very fruitful for McCLIM development. We'd like to thank all contributors once more and we wish you all (and ourselves) that the next year will be at least as good as this one, a lot of joy, creativeness and Happy Hacking!

Sincerely yours,
McCLIM Development Team

As programmers, we often deal with the inheritance and the composition, especially since OOP is a dominating paradigm of programming (no matter if the central concept is the object or the function). Not so often we deal with the third type of connection, that is the Form and the phenomena which are merely a shadow mimicking it[1].

Let us talk about sheets. The sheet is a region[2] with an infinite resolution and potentially infinite extent on which we can draw. Sheets may be arranged into hierarchies creating a windowing system with a child-parent relation. Sheet is the Form with no visual appearance. What we observe is an approximation of the sheet which may be hard to recognize when compared to other approximations.

[1]: Theory of forms.

[2]: And many other things.

Physical devices

Sheet hierarchies may be manipulated without a physical medium but to make them visible we need to draw them. CLIM defines ports and grafts to allow rooting sheets to a display server. medium is defined to draw sheet approximation on the screen[3].

How should look a square with side length 10 when drawn on a sheet? Since it doesn't have a unit we can't tell. Before we decide on a unit we choose we need to take into consideration at least the following scenarios:

1. If we assume device-specific unit results may drastically differ (density may vary from say 1200dpi on a printer down to 160dpi on a desktop[4]!. The very same square could have 1cm on a paper sheet, 7cm and 10cm on different displays and 200cm on a terminal. From the perspective of a person who uses the application, this may be confusing because physical objects don't change size without a reason.

2. Another possible approach is to assume the physical world distances measured in millimeters (or inches if you must). Thanks to that object will have a similar size on different mediums. This is better but still not good. We have to acknowledge that most computer displays are pixel based. Specifying distances in millimeters will inevitably lead to worse drawing quality[5] (compared to the situation when we use pixels as the base unit). Moreover, conversion from millimeter values to the pixel will most inevitably lead to work on floats.

3. Some applications may have specific demands. For instance, application is meant to run on a bus stop showing arrival times. Space of the display is very limited and we can't afford approximation from the high-density specification (in pixels or millimeters) to 80x40 screen (5 lines of 8 characters with 5x8 dots each). We need to be very precise and the ability to request a particular unit size for a sheet is essential. Of course, we want to test such application on a computer screen.

I will try to answer this question in a moment. First, we have to talk about the CLIM specification and limitations imposed by McCLIM's implementation of grafts.

[3]: See some general recommendations.

[4]: Technically it should be PPI, not DPI (pixels per inch).

[5]: Given programmer specifies sheet size in integers (like 100x100).

Ports, grafts, and McCLIM limitations

If a port is a physical connection to a display server then graft is its screen representation. The following picture illustrates how the same physical screen may be perceived depending on its settings and the way we look at it.

As we can see graft has an orientation (:default starts at the top-left corner like a paper sheet and :graphics should start at the bottom left corner like on the chart). Moreover, it has units. Currently, McCLIM will recognize :device, :inches, :millimeters and :screen-sized [11].

That said McCLIM doesn't implement graft in a useful way. Everything is measured in pixels (which :device units are assumed to be) and only the :default orientation is implemented. By now we should already know that pixels are a not a good choice for the default unit. Also, programmer doesn't have means to request unit for a sheet (this is the API problem).

[11]: Screen size unit means that the coordinate [1/2, 1/2] is exactly in the middle of the screens.

Physical size and pixel size compromise

We will skip the third situation, for now, to decide what unit should we default to. There are cognitive arguments for units based on a real-world distance but there are also and practical reasons against using millimeters – poor mapping to pixels and the fact that CLIM software which is already written is defined with the assumption that we operate on something comparable to a pixel.

Having all that in mind the default unit should be device-independent pixel. One of McCLIM long-term goals is to adhere to Material Design guidelines – that's why we will use dip[6] unit. 100dp has absolute value 15.875mm. On 160dpi display it is 100px, on 96dpi display it is 60px, on 240dpi display it is 150px etc. This unit gives us some nice characteristics: we are rather compatible with the existing applications and we preserving absolute distances across different screens.

[6]: Density-independent pixel.

How to draw a rectangle on the medium

Application medium may be a pixel-based screen, paper sheet or even a text terminal. When the programmer writes the application he operates on dip units which have absolute value 0.15875mm. It is the McCLIM responsibility to map these units onto the device. To be precise each graft needs to hold an extra transformation which is applied before sending content to the display server.

Now we will go through a few example mappings of two rectangle borders[7] drawn on a sheet. The violet rectangle coordinates are [5,5], [22,35] and the cyan rectangle coordinates are [25,10], [30,15].

• MDPI display device units are dip and they match native units of our choosing. No transformation is required.

• Some old displays have density 72PPI. Not all coordinates map exactly to pixels - we need to round them[3]. Notice that border is thicker and that proportions are a little distorted. On the other hand despite a big change in resolution size of the object is similar in real-world values.

Windows Presentation Foundation declared 96PPI screen's pixel being the device-independent pixel because such displays were pretty common on desktops. Almost all coordinates map perfectly on this screen. Notice the approximation of the right side of the violet rectangle.

• Fact that the screen has higher density doesn't mean that coordinates mapping perfectly on a lower density screen will map well to a higher density one. Take this HDPI screen. Almost all coordinates are floats while on the MDPI display they had all integer values.

• Higher resolution makes rectangles look better (borderline is thinner and distortions are less visible to the eye). Here is XXXHDPI:

• Some printers have a really high DPI, here is imaginary 2560 DPI printer. Funnily enough its accuracy exceeds our screen density so the red border which is meant to show the "ideal" rectangle is a little off (it's fine if we scale the image though).

• Until now we've seen some screens with square pixels (or dots). Let's take a look at something with a really low density - a character terminal. To make the illustration better we assume an absurd terminal which has 5x8 DP per character (too small to be seen by a human eye). Notice that the real size of the rectangles is still similar.

It is time to deal with graphics orientation (Y-axis grows towards the top). An imaginary plotter with 80DPI resolution will be used to illustrate two solutions (the first one is wrong!). Knowing the plotter screen height is important to know where we start the drawing.

• Graft reverts Y axis and sends the image to the plotter. Do you see what is wrong with this picture? We have defined both rectangles in default orientation, so our drawing should look similar disregarding the medium we print on. We do preserve the real size but we don't preserve the image orientation – cyan rectangle should be higher on the plot.

• Correct transformation involves reverting Y axis and translating objects by the screen height. See the correct transformation (on 80DPI and on MDPI plotter).

[7]: the Ideal border is composed of lines which are 1-dimensional objects which doesn't occupy any space. Red border in drawings marks "ideal" object boundaries. Points are labeled in device units (with possible fractions).

Sheets written with a special device in mind

There is still an unanswered question - how can we program applications with a specific device limitations in mind? As we have discussed earlier default sheet unit should be dip and default sheet orientation is the same as a paper sheet's[8].

Writing an application for a terminal is different than writing an application for a web browser. The number of characters which fit on the screen is limited, drawing shapes is not practical etc. To ensure that the application is rendered correctly we need a special kind of sheet which will operate on units being characters. Take a look at the following example.

The first sheet base unit is a character of a certain physical size. We print some information on it in various colors. Nothing prevents us from drawing a circle[9] but that would miss the point. We use a character as a unit because we don't want rounding and approximation. Background and foreground colors are inherited.

The second sheet base unit is dip. It has two circles, solid grey background and a few strings (one is written in italic).

Ideally, we want to be able to render both sheets on the same physical screen. The graft and the medium will take care of the sheet approximation. The effect should look something like this.

The first screen is a terminal. Circles from the second sheet are approximated with characters and look ugly but the overall effect resembles the original application. Proportions are preserved (to some degree). We see also the terminal-specialized sheet which looks exactly as we have defined it.

The second screen is mDPI display. The second sheet looks very much like something we have defined. What is more interesting though is our first sheet – it looks exactly the same as on the terminal.

[8]: Providing means to change defaults requires additional thought and is a material for a separate chapter. McCLIM doesn't allow it yet.

[9]: As we have noted sheets are ideal planar spaces where line thickness is 0 and there is nothing preventing us from using fractions as coordinates.

Port and graft protocols

Now we know what we want. Time to think about how to achieve it. Let me remind you what kind of objects we are dealing with:

• Port is a logical connection to a display server. For instance, it may contain a foreign handler which is passed to the external system API. It is responsible for the communication – configuring, writing to and reading[10] from a device, we are connected to.

• Graft is a logical screen representation on which we draw. It is responsible for all transformations necessary to achieve the desired effect on the physical screen. The same port may have many associated grafts for applications with different units and orientations.

• Medium is a representation of the sheet on a physical device. Sheet is the Form which is a region and may be drawn – it doesn't concern itself with physical limitations.

In the next post I will show how to implement the port and the graft (and a bit of the medium) for the charming backend. I will cover only bare minimum for mediums important to verify that graft works as expected. Complete medium implementation is the material for a separate post.

[10]: Sometimes we deal with devices which we can't take input from – for instance a printer, a PDF render or a display without other peripherals.

Due to a long period of time, the number of changes is too big to list in full detail and we will thus note only major changes made during the last eleven iterations (though many important changes were done before that). For more information please check out previous iteration reports on McCLIM blog, git log and the issue tracker. We'd like to thank all present and past contributors for their time, support and testing.

• Bug fix: tab-layout fixes.
• Bug fix: formatting-table fixes.
• Bug fix: scrolling and viewport fixes and refactor.
• Feature: raster image draw backend extension.
• Feature: bezier curves extension.
• Feature: new tests and demos in clim-examples.
• Feature: truetype rendering is now default on clx.
• Feature: additions to region, clipping rectangles and drawing.
• Feature: clim-debugger and clim-listener improvmenets.
• Feature: mop is now done with CLOSER-MOP.
• Feature: threading is now done with BORDEAUX-THREADS.
• Feature: clx-fb backend (poc of framebuffer-based backend).
• Feature: assumption that all panes must be mirrored has been removed.
• Cleanup: many files cleaned up from style warnings and such.
• Cleanup: removal of PIXIE.
• Cleanup: removal of CLIM-FFI package.
• Cleanup: changes to directory structure and asd definitions.
• Cleanup: numerous manual additions and corrections.
• Cleanup: broken backends has been removed.
• Cleanup: goatee has been removed in favour of Drei.
• Cleanup: all methods have now corresponding generic function declarations.

We also have a bounty program financed with money from the fundraiser. We are grateful for financial contributions which allow us to attract new developers and reward old ones with bounties. Currently active bounties (worth $2650) are available here.

As Imbolc marks the beginning of spring we hope this release will be one of many in the upcoming future.

After three months of work, we are happy to present this progress report.

Some highlights for this iteration:

• numerous bug fixes (and some regressions introduced)
• graphics state has been factored into its own class hierarchy
• clipping region may be an arbitrary region (not only a rectangle)
• draw-design works correctly now for ellipses and patterns
• major region code improvements including better coverage of region-intersection, region-contains-position-p, support for unsupported yet pairs (i.e standard-rectangle vs bounding-rectangle, standard-ellipse vs standard-rectangle and many others)
• complete implementation of ellipsis with limited angles and rotations (region code and rendering)
• numerous UX improvements for text-field, describe, drag-and-drop, scrolling, compose-space, resize-sheet, clim-debugger and listener
• :pane option in define-application-frame is more useful now
• clim-formatting-table code major refactor. One of the results is that borders may wrap table cells, columns and rows appropriately
• drawing tests cleanups, improvements, and additions.

Some work has been done to improve CLX library. We have added a test suite and fixed its working on CCL (which had some problems with it). Thanks to that McCLIM works on CCL way faster than before (around 3x – subjective). We were also able to make CLIM-TOS (Franz's CLIM 2 fork) start simple applications on CCL and SBCL. I'm mentioning this because some people are not fond of McCLIM license and fact that we have two FOSS implementations of the same standard may be yet another good reason to start writing applications in CLIM.

Bounties:

All McCLIM bounties (both active and already solved) may be found here. Default bounty expiration date is 6 months after publishing it (a bounty may be reissued after that time period).

Bounties solved this iteration:

• [$100] drag-test demo: dragging square to the empty position invokes the debugger
• [$100] Text field pane height is too small, clipping the bottom of characters

Active bounties ($1600):

• [$300] Listener: repl blocks when long process runs
• [$500] Windows Backend
• [$400] Fix Beagle backend
• [$150] When flowing text in a FORMATTING-TABLE, the pane size is used instead of the column size
• [$150] clx: input: english layout
• [$100] Add PDF file generation (PDF backend)

Our current financial status is $2014 for bounties and $288 recurring monthly contributions from the supporters (thank you!).

Suggestions as to which other issues should have a bounty on them are appreciated and welcome. Please note that Bountysource has a functionality "Suggest an Issue" which may be found on the bounties page. If you would like to work on an issue that is not covered by the existing bounties, feel free to suggest a new bounty.

During this quarter we have noticed more interest in learning CLIM and developing applications with it among fellow Common Lisp programmers.

If you have any questions, doubts or suggestions – please contact me either by email (daniel@turtleware.eu) or on IRC (my nick is jackdaniel). McCLIM developers and users hang out on #clim IRC channel on Freenode.

Sincerely yours,
Daniel Kochmański

Here are some screenshots made in the meantime which are related to the changes:

We have many important improvements since the last iteration and even more work is pending. I want to apologise for this late progress report – it has been almost three months since the last update. I'll try to improve in this regard.

Some highlights for this iteration:

• various utilities have been replaced with alexandria equivalents
• distinct frames don't shadow *accelerator-gestures* of their children
• accepting-values refinements - better handling of errors and return values
• refactor and small fixes of the recording implementation code
• refinements in class hierarchy for streams, medium and graphics state
• slider from 30.4.5 spec section has been implemented
• scrolling implementation refinements
• tab-layout extension refactor
• improvements related to drei text editing substrate
• user manual refinements and improvements of its building scripts
• improvements related to the PDF backend
• MOP code has been ported to use closer-mop portability layer
• numerous editorial fixes in bundled specification sources
• improvements to format-graph-from-roots
• better Unicode support in CLX for frame title
• general code base cleanup to decrease number of warnings during compilation
• transparency handling in CLX backend and alpha channel support in images
• small Listener improvements (bug fixes and cleanups)

We want to thank everybody who has contributed to the project (by improving the code base, discussions, issue reporting, providing advice and suggestions, monetary contributions etc). We are especially grateful to the following people: Nisar Ahmad, Alastair Bridgewater, John Carroll, Cyrus Harmon, Philipp Marek, Elias Mårtenson, Piotr Mieszkowski, Jan Moringen, Nick Patrick, Alessandro Serra and last but not least Robert Strandh.

Bounties:

All McCLIM bounties (both active and already solved) may be found here. Default bounty expiration date is 6 months after publishing it (a bounty may be reissued after that time period).

Bounties solved this iteration:

• [$300] Replace MOP things with closer-mop portability layer
• [$100] Keystroke accelerators may shadow keys even if inactive

Active bounties ($1800):

• [$100] drag-test demo: dragging square to the empty position invokes the debugger (new)
• [$100] Text field pane height is too small, clipping the bottom off characters (new)
• [$300] Listener: repl blocks when long process runs (new)
• [$500] Windows Backend
• [$400] Fix Beagle backend
• [$150] When flowing text in a FORMATTING-TABLE, the pane size is used instead of the column size
• [$150] clx: input: english layout
• [$100] Add PDF file generation (PDF backend)

Our current financial status is $1089 for bounties and $264 recurring monthly contributions from the supporters (thank you!).

I have been asked a question about who decides which issues have bounties on them and how the reward amounts are decided. If anyone has been wondering about the same here goes the answer: issues and prices are based on my subjective opinion indicated by problems users encounter and what I consider being worth putting bounty on it. Note though, that I'm open to suggestions (see the next paragraph). I hope that despite some potential doubts the community is generally satisfied with the progress and decisions we make. If there is some lack of transparency, please let me know what you want to know and I'll do my best to help.

Suggestions as to which other issues should have a bounty on them are appreciated and welcome. Please note that Bountysource has a functionality "Suggest an Issue" which may be found on the bounties page. If you would like to work on an issue that is not covered by the existing bounties, feel free to suggest a new bounty.

If you have any questions, doubts or suggestions – please contact me either by email (daniel@turtleware.eu) or on IRC (my nick is jackdaniel). McCLIM developers and users hang out on #clim IRC channel on Freenode.

Sincerely yours,
Daniel Kochmański

McCLIM code is getting better on a weekly basis depending on developer time. We are happy to see the project moving forward.

Some highlights for this iteration:

• Scigraph code cleanup and bug fixes,
• Bezier curves improvements,
• PostScript and PDF improvements,
• CLX-fb and mcclim-renderer speed improvements and refactor,
• various code cleanups from unused and broken constructs,
• editorial corrections to the CLIM 2 specification sources we bundle with McCLIM

Moreover many bug fixes have been proposed and merged into the codebase.

All McCLIM bounties (both active and already solved) may be found here. Default bounty expiration date is 6 months after posting it (a bounty may be reissued after that time period).

To answer recurring requests for native Windows and OSX support, we have posted bountes for finishing the Windows backend and fixing the OSX backend. Moreover, to improve portability a bounty for closer-mop support has been posted.

Bounties solved this iteration:

• [$100] Caps lock affects non-alphabetic keys.

Active bounties ($1700):

• [$500] Windows Backend (new).
• [$400] Fix Beagle backend (new).
• [$300] Replace MOP things with closer-mop portability layer (new).
• [$150] When flowing text in a FORMATTING-TABLE, the pane size is used instead of the column size.
• [$150] clx: input: english layout.
• [$100] Add PDF file generation (PDF backend).
• [$100] Keystroke accelerators may shadow keys even if inactive.

Our current financial status is $800 for bounties and $267 recurring monthly contributions from the supporters (thanks!).

Suggestions as to which other issues should have a bounty on them are appreciated and welcome. Please note that Bountysource has a functionality "Suggest an Issue" which may be found on the bounties page. If you would like to work on an issue that is not covered by the existing bounties, feel free to suggest a new bounty.

If you have any questions, doubts or suggestions – please contact me either by email (daniel@turtleware.eu) or on IRC (my nick is jackdaniel).

Sincerely yours,
Daniel Kochmański