.. @+leo-ver=5-thin
.. @+node:ekr.20031218072017.329: * @file ../doc/leoNotes.txt
.. @@language rest
.. @@killbeautify
.. @+all
.. @+node:ekr.20131005214621.16081: ** Notes
.. @+node:ekr.20090202064534.4: *3*  Your mission, should you choose to accept it
@language rest

.. @+node:ekr.20100223100750.5843: *4* Original post by Robin Dunn
Here are the emacs features that I use very often that any editor would
need to have in order for me to switch.  I've seen some editors with
some of these, but none with all unless it is an emacs clone.  I'll
leave out the obvious things like platform independence, good syntax
highlighting, calltips or auto-completion.  Also, these features are
just dealing with the code editor portion of the app, if it is more than
that (like a full IDE) then some of these things may or may not apply to
the non code editor parts:

* (done) Python should be just one of the languages that this editor supports,
not the primary target.  I spend as much time in C/C++ as I do Python,
and my editor of choice needs to help me with C/C++ coding just as much
as it does with Python.  So some sort of support for calltips and
auto-completion would be marvelous, and also being able to act as a
front-end for gdb since I currently use emacs for that most of the time.

* (done) Absolutely every feature or action must be able to be done with just
the keyboard.  Moving the hand back and forth to the mouse wastes time,
breaks concentration and contributes to RSI.  Multi-key sequences are
fine as long as they are grouped in a logical fashion.  For example in
emacs all of the version control features are accessible via the
Ctrl-x,v sequence plus one more letter.

* (done) Incremental search, both forward and reverse, and wrapping around
after you've reached the end or the beginning of the document.  I like
to have the same key to start the search and also do a search-next after
you've typed all the characters you are searching for, and also to have
backspace go back one search position and/or remove one character from
the search text.

* (done) Multiple top level windows, and able to show any buffer in any TLW,
including those that are already displayed in another TLW.  Of course
there should be key-bindings available for opening a new TLW, cycling
forward and backward through the buffer list, and a way to select a
buffer from a popup list of buffer/file names.

* (to be improved) The Kill-Ring.  For those of you that have never used an emacs-like
editor it works like this:  There is a collection of the N previous
blocks of text that have been cut or copied (in emacs 'cut' == 'kill'
more or less)  When I do a yank (paste) it uses the last thing put in
the kill-ring.  If I then immediately use another key-binding then it
replaces that pasted text with the next item in the kill ring, and so on
until I eventually wrap around get back to the first one in the ring, or
I do some other command or move the cursor somewhere else.

* (done) Registers.  A text snippet can be copied into a register, which is
like the kill ring except you refer to each one by name, where the names
are 'a' through 'z'.  You can also append to a register that already has
text in it, and you can paste the contents of a register into the
document at the current cursor location.

* (done) Able to have selections be either a stream of characters or a
rectangle.  A stream selection is like what you have in all text
editors, it starts from position a on line N and continues forward or
back to position b on line M and includes all the characters in between.

  A rectangle selection is all the characters between position a and b
on lines N to M.  In other words, it has width and height and it might
be something like positions 5 through 10 on lines 20 to 25.  Cutting or
deleting a rectangle removes the text in the rectangle and shifts any
text to the right of the rectangle over.  It does not remove any lines
although they may end up being empty.  Pasting a rectangle inserts the
new text with the upper-left of the rectangle at the current cursor
position, shifts existing text to the right if needed, and fills with
spaces on the left if a line affected by the paste is not long enough.
New lines are not added unless the file needs to be extended to
accommodate the rectangle paste.  Rectangles can also be put into registers.

* (to be improved) Good keystroke macro recording and the ability to save and load
keystroke macros, and the ability to assign a key-binding to a saved
recorded macro. Any time I need to make the same edits to a bunch of
lines or groups of lines I'll record doing it on the first one including
the keystrokes needed to reposition for the next line, and then stop
recording and then it's just one keystroke to replay the keystrokes for
every other line that needs it done.  I record, use and throw away up to
a dozen or so macros per day.

* (done, and better than asserted) If you must have a toolbar make it optional
and keep it simple. Toolbars require the mouse and the goal is to keep the hand
off the mouse as much as possible.

* (done) Similarly, avoid using popup dialogs whenever possible.  This includes
things like the file dialog.  I don't mind seeing the file dialog if I
select a menu item, because most likely my hand is already on the mouse,
but the rest of the time I just want to hit a key, type a path name
(with tab-completion to help find stuff, up/down keys to cycle through
past selections) and press enter.  So I would prefer this editor to have
something like emacs' minibuffer, or the QuickFind panel in Firefox.  In
other words, when there is something you would normally use a dialog for
just create a small panel that rolls up from the bottom of the frame,
put the keyboard focus there, perhaps do stuff in the main buffer as
they are typing if appropriate, and then when the user is done the panel
rolls out of sight again and keyboard focus is restored to their active
buffer.  This can be done for file open/saves, search & replace,
specifying build or grep commands (see next item) choosing to execute
some editor function by name that may not have some key-binding yet (see
item after next) etc.

* (done, with user @commands)

Flexible build/grep commands.  Emacs handles both of these in almost
the same way so I'll list them together here.  I hit a key and am
presented with either the default, or the most recently used compile or
grep command.  I can edit the command or use the up/down arrows to
select previous commands that I've used.  I then hit enter and emacs
runs the command putting the output in an editor buffer.  There is a key
I can hit to kill the compile if needed.  It then parses the output and
there is a key I can use to find the file listed in the compile or grep
output, load it, and position the cursor on the reported line.  (This
can even be done while the compile/grep is still running.)

* (done) For access to editor commands/functionality that may not be bound to a
keystroke it's real nice to have the ability to hit a key, type the
command name, press enter and then it's done.  This can also allow for
commands that might need to prompt for parameters, be interactive, etc.
  All editor commands should be named and can be bound to keys by name
or executed by name in this way.

* (done) def aproposFindCommands (self, event=None):

    '''Prints a discussion of of Leo's find commands.'''

    << define s >>

    self.c.putApropos(s)
 search.  Emacs has support for regular expression search modes
for all of the search types, incremental search, search/replace,
although I don't use it that much.

* (done, or not needed, depending on your point of view)
Multi-file search and replace.  Be able to select files interactively,
or by wildcard, or both.  Enter search string, or regex, and replace
text.  The editor loads each file and does the search, allowing you to
choose for each one whether to do the replacement, or replace all.

* If it is a full IDE it would be nice to have a way to start just the
code editor portion for quick edits.

Things that would be nice to have, but that I could live without:

(All of these things can be done easily with @command)

* Interactive diffs, merges and applying of patches.

* Able to be a front-end for gdb.

* Able to be a front-end for CVS, SVN, etc.

* (done) Be able to run shell commands, or the shell itself in an editor buffer.

* (easy) have a built-in psychotherapist or be able to play towers of hanoi.  ;-) 
.. @+node:ekr.20100223100750.5842: *4* Post to pyxides, 2010/02/23
http://groups.google.com/group/pyxides

Robin Dunn's post, reproduced at:
http://groups.google.com/group/leo-editor/browse_thread/thread/4f76a0f57759aba
continues to be one of the benchmarks for Leo.

Leo 4.7 went out the door today.  It contains many important
improvements, but few directly related to Robin's important post.
That doesn't mean Robin's post is irrelevant, but it does mean that
other considerations were more relevant :-)  In particular, Leo passes
all unit tests with Python 2.6 and Python 3.1.

Leo 4.8 will concentrate on better support for vim-like bindings.  As
a happy side effect, this will make Leo compliant with almost all
unfinished aspects of Robin's mission.

There are two major items from Robin's list that are incomplete in
Leo:

* The Kill-Ring.  Leo does have a kill ring.  The vim work will fix
discrepancies between how Leo, emacs and vim handle the kill ring.

* Good keystroke macro recording and the ability to save and load
keystroke macros.  This happen as part of support for vim's "dot"
command.  To some extent, Leo's execute-script command compensates for
wimpy macro support, but I'd like to do better.

The following could be done easily using Leo's @command or @button
features.  There doesn't seem to be much demand for them in Leo, but
I'll list them here for completeness.

- Flexible build/grep commands.
- Interactive diffs, merges and applying of patches.
- Able to be a front-end for gdb. (Leo has a plugin to do this).
- Able to be a front-end for CVS, SVN, etc.
- Have a built-in psychotherapist or be able to play towers of
hanoi.  ;-)

As always, I invite you all to try Leo, and to ask for features that
would be important to you.

======

P.S. Leo does have auto-completion.  It will be improved in Leo 4.9.

.. @+node:ekr.20110616084347.14800: *4* Post to pyxides, 2011/07/10
http://groups.google.com/group/pyxides

On 2010/02/23 I commented about Leo 4.7 as it relates to Robin Dunn's post,
http://groups.google.com/group/leo-editor/browse_thread/thread/4f76a0f57759aba

A few weeks ago Leo 4.9 went out the door. Imo, this version of Leo has
accomplished the mission. Leo has all the important features that Leo's users
have requested. Yes, wishlist items remain. See:
https://bugs.launchpad.net/leo-editor/+bugs

None of these wish-list items interferes in any way with Leo's day-to-day
operation. Furthermore, many of Leo's essential features moot the need for more
traditional features.

For example, Leo 4.9 adds full support for macros. Recording, saving, editing
and retrieving macros is easier in Leo than in other editors because Leo stores
macros in @macro nodes, not external files. But few, if any, of Leo's users are
likely to use macros because Leo's @button nodes make all of Python's scripting
abilities easily available on a node-by-node or outline-wide basis.

It may be that Leo could benefit from some Emacs-like or vim-like features, but
that doesn't seem so likely.  Leo has many users who also use Emacs and vim, and
they seem happy enough :-)

Finally, Leo offers features that have no counterpart at all in editors like
Emacs and vim. For example, the rst3 command converts an outline to restructured
text. See: http://webpages.charter.net/edreamleo/rstplugin3.html Yes, one could
imagine an org-mode command that does this, but the fact is that Leo's outline
orientation has given it abilities possessed by no other editor or IDE.

I invite you to try Leo. If, after using Leo for
real work, you find you would like some new feature, then by all means ask.

Edward
.. @+node:ekr.20061116060847: *3* @url http://www.jhorman.org/wikidPad/
.. @+node:ekr.20150520135110.1: *3* autopep8 notes
.. @+node:ekr.20150520111038.1: *4* autopep8
@language rest
@wrap

https://github.com/hhatto/autopep8#usage

Suppress these:

E301 - Add missing blank line.
E302 - Add missing 2 blank lines.

Create local config file, wherever that is!
.. @+node:ekr.20150520065100.1: *4* About settings
Existing settings:
    @bool tidy_add_blank_lines_around_comments = False ###
    @bool tidy_double_quoted_strings = False
    @bool tidy_java_style_list_dedent = True
        ### Essential for continued def args.
    @bool tidy_keep_blank_lines = True ###
    @bool tidy_left_adjust_docstrings = False
    @int  tidy_lines_before_split_lit = 2
    @int  tidy_max_seps_func_def = 3

New settings:
    @bool tidy_blank_line_after_docstring = False 
    @bool tidy_blank_line_after_func_def = True
    @bool tidy_leo_call_continuation = False
    @bool tidy_spaces_after_docstring = False
    @int tidy_col_limit = 75
    @int tidy_max_seps_func_def = 20
.. @+node:ekr.20150520085848.1: *4* 2to3 log
@language rest
@wrap

c:\leo.repo\leo-editor\leo\external>c:\python27\python.exe c:\Python27\Tools\Scripts\2to3.py -w PythonTidy.py
RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
RefactoringTool: Skipping implicit fixer: set_literal
RefactoringTool: Skipping implicit fixer: ws_comma
RefactoringTool: Refactored PythonTidy.py
--- PythonTidy.py       (original)
+++ PythonTidy.py       (refactored)
@@ -119,7 +119,7 @@

 '''
 #@-<< docstring >>
-from __future__ import division
+
 VERSION = '1.23.1'
     # EKR: based on version 1.23, 2012 May 23
     # http://lacusveris.com/PythonTidy/PythonTidy-1.23.python
@@ -875,7 +875,7 @@
         self.end = len(self.lines) - 1
         return self
     #@+node:ekr.20141010141310.18651: *4* next
-    def next(self):
+    def __next__(self):

         if self.ndx > self.end:
             raise StopIteration
@@ -893,7 +893,7 @@
     def readline(self):

         try:
-            result = self.next()
+            result = next(self)
         except StopIteration:
             result = ''
         return result
@@ -1219,14 +1219,14 @@
             """
             try:
                 while True:
-                    prev_item = lines.next()
+                    prev_item = next(lines)
                     yield prev_item
                     prev_token_type, prev_token_string, prev_start, prev_end, \
                         prev_line = prev_item
                     if prev_token_type in [tokenize.STRING]:
                         on1 = True
                         while True:
-                            next_item = lines.next()
+                            next_item = next(lines)
                             yield next_item
                             next_token_type, next_token_string, next_start, \
                                 next_end, next_line = next_item
@@ -1254,8 +1254,8 @@
         lines = merge_concatenated_strings(lines)
         for token_type, token_string, start, end, line in lines:
             if False and DEBUG:
-                print (token.tok_name[token_type], token_string, start, end,
-                       line)
+                print((token.tok_name[token_type], token_string, start, end,
+                       line))
             self.max_lineno, scol = start
             erow, ecol = end
             if token_type in [tokenize.COMMENT, tokenize.NL]:
@@ -1826,7 +1826,7 @@
             self.str = g.toEncodedString(str_)
         else:
             self.str = str_
-            if isinstance(self.str, unicode):
+            if isinstance(self.str, str):
                 pass
             elif not RECODE_STRINGS:
                 pass
@@ -5846,7 +5846,7 @@
         result = NodeWith(indent, lineno, node.expr, node.vars, node.body)
     elif isinstance_(node, 'Yield'):
         result = NodeYield(indent, lineno, node.value)
-    elif isinstance(node, basestring):
+    elif isinstance(node, str):
         result = NodeStr(indent, lineno, node)
     elif isinstance(node, int):
         result = NodeInt(indent, lineno, node)
RefactoringTool: Files that were modified:
RefactoringTool: PythonTidy.py
RefactoringTool: Warnings/messages while refactoring:
RefactoringTool: ### In file PythonTidy.py ###
RefactoringTool: Line 3415: cannot convert map(None, ...) with multiple arguments because map() now truncates to the sho
rtest sequence
RefactoringTool: Line 3415: cannot convert map(None, ...) with multiple arguments because map() now truncates to the sho
rtest sequence

.. @+node:ekr.20150520135211.1: *4* Unused code in ppp.python_tidy
    # refs = []
    # for line in g.splitLines(s1):
        # i = line.find('<<')
        # j = line.find('>>')
        # if -1 < i < j:
            # ref = line[i:j]
            # refs.append(ref)
            # ref2 = ref.replace('<<','# < <')
            # s1 = s1.replace(ref,ref2)
.. @+node:ekr.20031218072017.365: *3* How to...
.. @+node:ekr.20051203084725: *4* How to expand java .jar files
- Put whatever.jar in c:\prog
- cd: c:\prog
- jar xvf whatever.jar
.. @+node:ekr.20031218072017.384: *4* How to export syntax colored code preserving colors
Scite has the option to "Export as html" and "export as rtf", and it will be
full of colour and fonts - and you can define them in properties, so it will be
the same as during editing.
.. @+node:ekr.20060331094112: *4* How to generate keys using putty
To generate a SSH key using PuTTY:

Execute c:\"Program Files"\tortoiseCVS\PUTTYGEN.EXE

Select "SSH2 DSA", within the "Parameters" section.

Click on the "Generate" button. Follow the instruction to move the mouse over
the blank area of the program in order to create random data used by PUTTYGEN to
generate secure keys. Key generation will occur once PUTTYGEN has collected
sufficient random data.

Enter edream@cvs.sourceforge.net for the key comment (depends on what host the
key is for)

(Omit) Enter the desired passphrase in the "Key passphrase" and "Confirm passphrase"
fields. If the key will be used for automation of operations (i.e. as part of a
script), you may choose to omit this step from the key generation process.

Click on the "Save private key" button. Use the resulting dialog to save your
private key data for future use. You may use a filename such as
"SourceForge-Shell.ppk" or "SourceForge-CF.ppk". The .ppk extension is used for
PuTTY Private Key files.

Go to the SSH key posting page on the SourceForge.net site: http://sourceforge.net/account/

Copy your public key data from the "Public key for pasting into OpenSSH
authorized_keys2 file" section of the PuTTY Key Generator, and paste the key
data to the provided form on the SourceForge.net site. Click on the "Update"
button to complete the posting process.

Exit the PuTTY Key Generator (PUTTYGEN).

Key data sync to hosts from the SourceForge.net site occurs on regular
intervals. Your key data will be synchronized to the designated servers (either
shell and CVS, or the Compile Farm) after a short delay.
.. @+node:ekr.20100904134301.8336: *4* How to generate pdf on Linux
El 01/05/09 15:12, Ville M. Vainio escribió:

- make latex
- cd _build/latex
- make all-pdf
.. @+node:ekr.20031218072017.385: *4* How to Increase environment space
To increase the size of environment space, add the following to config.sys:

shell=C:\windows\command\command.com /p:4096

Notes:

1. The path C:\windows\command\command.com may vary.
Check you system for the location of command.com.

2. This works for versions of Windows prior to Me.
On Me you set the registry somehow.
No information on XP.
.. @+node:ekr.20101004092958.6050: *4* How to make codewise work
http://groups.google.com/group/leo-editor/browse_thread/thread/ac3f8789010c882e/a1558a10eb8537c0?lnk=gst&q=codewise#a1558a10eb8537c0

1. Make sure you have exuberant ctags (not just regular ctags)
installed.  It's an Ubuntu package, so easy if you're using Ubuntu.

2. Install Ville's python module "codewise".  This is a small module on
which the Leo plugin relies.

   bzr branch lp:codewise
   cd codewise
   sudo python setup.py install

3. You need a recent trunk version of leo to get the plugin which uses
the above module.

4. Enable the plugin by putting "codewisecompleter.py" on an
uncommented line in your @enabled-plugins @settings node.

5. On the command line:

if you have an existing ~/.ctags for some reason, and it's nothing you
need to keep:

  rm ~/.ctags

then

  codewise setup
  codewise init
  codewise parse .../path/to/leo/  # assuming you want completion on
                                   # leo code
  codewise parse .../some/other/project/

Then, after restarting leo if necessary, type

c.op<Alt-0> in the body editor to find all the c. methods starting
with 'op' etc.

Nice work Ville, thanks.

==================

Thanks for this, I hope others will take a stab at it as well, given
sane instructions (I burned my free cycles frantically coding this
thing and neglected the all-important HOWTO). This is important
because functional completion is the single most important thing still
missing from Leo. Or, well, was ;-).

Especially the presentation part (QCompleter) needs some care, so you
can operate it from your keyboard alone. It should probably be moved
to core (qtgui, perhaps leoQTextEditWIdget), so codewise completer can
just invoke w.complete(list_of_completions) that will bring up the
QCompleter popup.

> Then, after restarting leo if necessary, type

> c.op<Alt-0> in the body editor to find all the c. methods starting
> with 'op' etc.

Also, try the explicit declarations:

# w : SomeClass

w.<alt+0>

And self.<alt+0> 
.. @+node:ekr.20091217112515.6070: *4* How to make the codewise completer work
http://groups.google.com/group/leo-editor/browse_thread/thread/ac3f8789010c882e

Ville's completer is working and very cool, here are instructions for
making it go.  They're like the instructions Ville gave, only usable ;-)

1. (done) Make sure you have exuberant ctags (not just regular ctags)
installed.  It's an Ubuntu package, so easy if you're using Ubuntu.

2. (done) Install Ville's python module "codewise".  This is a small module on
which the Leo plugin relies.

   bzr branch lp:codewise
   cd codewise
   sudo python setup.py install

3. (done) You need a recent trunk version of leo to get the plugin which uses
the above module.

4. (done) Enable the plugin by putting "codewisecompleter.py" on an
uncommented line in your @enabled-plugins @settings node.

5. On the command line:

if you have an existing ~/.ctags for some reason, and it's nothing you
need to keep:

  rm ~/.ctags

then

  codewise setup
  codewise init
  codewise parse .../path/to/leo/  # assuming you want completion on
                                   # leo code
  codewise parse .../some/other/project/

Then, after restarting leo if necessary, type

c.op<Alt-0> in the body editor to find all the c. methods starting
with 'op' etc.


===== Ville's response

Especially the presentation part (QCompleter) needs some care, so you
can operate it from your keyboard alone. It should probably be moved
to core (qtgui, perhaps leoQTextEditWIdget), so codewise completer can
just invoke w.complete(list_of_completions) that will bring up the
QCompleter popup.

> Then, after restarting leo if necessary, type

> c.op<Alt-0> in the body editor to find all the c. methods starting
> with 'op' etc.

Also, try the explicit declarations:

# w : SomeClass

w.<alt+0>

And self.<alt+0>
.. @+node:ekr.20091217112515.6069: *5* Others posts
1. codewisecompleter.py now completes by explicit type hints (as seen in
screenhots). p, c also work, as does 'self'.

self works by scanning for parent headlines looking for "class Foo"

Work remains for presentation part (it's mouse only now) but Edward
will probably do it :-).

2. > Would codewise work outside of leo, as stand-alone plugin for a text
> editor?

Yes, currently Leo uses it as an external program ("codewise m
MyClass" dumps the methods in MyClass to stdout).

Someone just has to write the vim integration plugin (or whatever they
call it). OTOH, vim already has "pysmell" and the likes that do the
same thing.

==============

The version of codewise completer that works with Tk is now on trunk.
.. @+node:ekr.20091217112515.6071: *5* plugin docs
- You need to create ctags file to ~/.leo/tags. Example::

    cd ~/.leo
    ctags -R /usr/lib/python2.5 ~/leo-editor ~/my-project

- Enter text you want to complete and press alt+0 to show completions
  (or bind/execute ctags-complete command yourself).

Attempting to complete 'foo->' is useless, but 'foo->ba' will work (provided you
don't have 2000 functions/methods starting with 'ba'. 'foo->' portion is ignored
in completion search.
.. @+node:ekr.20031218072017.386: *4* How to remove cursed newlines: use binary mode
teknico ( Nicola Larosa ) 
 RE: Removing '\r' characters?   
2002-09-16 14:27  
> I am plowing through old bug reports, and I found the following, from whom 
> I don't know: 

That's from me, *again*. You are kindly advised to stop forgetting the attribution to all my bug reports. ;^) 

>> - Source files still have the dreaded \r in them. Why don't you switch 
>> to \n only, once and for all, and live happily ever after? ;^) 

> I sure whould like to do that, and I'm not sure how to do this. All 
> versions of the read code attempt to remove '\r' characters, and all 
> versions of the write code write '\n' only for newlines. 

Sorry for being a bit vague, I was talking about the Leo source files themselves. I don't know what you use to edit them, ;^))) but in version 3.6 they still have \r\n as end-of-line. 

If Leo itself does not solve the problem, may I suggest the 
Tools/scripts/crlf.py script in the Python source distibution? It's nice and simple, and skips binary files, too. That's what I use every time I install a new version of Leo. :^) 

.. @+node:ekr.20031218072017.387: *5* The solution
Under unix, python writes "\n" as "\n"; under windows, it writes it as "\r\n". The unix python interpreter ignores trailing "\r" in python source files. There are no such guarantees for other languages. Unix users should be able to get rid of the cosmetically detrimental "\r" either by running dos2unix on the offending files, or, if they're part of a .leo project, reading them into leo and writing them out again.  


By: edream ( Edward K. Ream ) 
 RE: Removing '\r' characters?   
2002-09-17 09:34  
Oh, I see. Thanks very much for this clarification. 

Just to make sure I understand you: the problem with '\r' characters is that: 

1. I am creating LeoPy.leo and LeoDocs.leo on Windows and 
2. People are then using these files on Linux. 

and the way to remove the '\r' characters: 

1. I could run dos2unix on all distributed files just before committing to CVS or making a final distribution or 
2. People could, say, do the following: 

Step 1: Read and Save the .leo files, thereby eliminating the '\r' in those files and 
Step 2: Use the Write @file nodes command on all derived files to clear the '\r' in those files. 

Do you agree so far? 

> Under unix, python writes "\n" as "\n"; under windows, it writes it as "\r\n". 

I am going to see if there is any way to get Python to write a "raw" '\n' to a file. I think there must be. This would solve the problem once and for all. 

Thanks again for this most helpful comment. 

Edward
.. @+node:ekr.20031218072017.388: *5* cursed newline answer
In 2.3 you can open files with the "U" flag and get "universal newline"
support: 

% python
Python 2.3a0 (#86, Sep 4 2002, 21:13:00) 
[GCC 2.96 20000731 (Mandrake Linux 8.1 2.96-0.62mdk)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open("crlf.txt")
>>> line = f.readline()
>>> line
'This is an example of what I have come to call the "cursed newline"
problem,\r\n'
>>> f = open("crlf.txt", "rU")
>>> line = f.readline()
>>> line
'This is an example of what I have come to call the "cursed newline" problem,\n'

.. @+node:ekr.20031218072017.389: *5* cursed newline answer 2
> You can open the file in 'binary' mode (adding 'b' to the mode string) and
> the file will contain '\r\n' on both platforms (and any other platforms.)

Nope. Exactly wrong. In 2.2 and those before, when files are opened in
*text* mode (no "b") then reading them will provide Unix-style line endings
(newline only). When you open files in binary mode then you see the bytes
stored in the file.

On Unix systems there's no difference in the contents of a file whether in
binary or text mode. On Windows a file is shorter by the number of carriage
returns. On the Mac I have no idea what they do. Probably just carriage
returns, to be different :-)

2.3 will be a bit more flexible about such mattrers.
.. @+node:ekr.20061023153133: *4* How to run patch
patch -p1 < patchfile
.. @+node:ekr.20050510071834: *4* How to use a temp file with pdb
@killcolor

http://sourceforge.net/forum/message.php?msg_id=3137690
By: nobody

I dont know if anyone has solved this for regular Leo, but in the JyLeo JythonShell,
when the user executes a script with Pdb it:
1. dumps the script in a tmp file system's tmp directory.
2. Executes pdb based off of that tmp file.

that way you get all the goodness that pdb can offer.
.. @+node:ekr.20141125112845.5: *3* Leo code notes
.. @+node:ekr.20131011050613.16839: *4* About autocompletion
.. @+node:ekr.20131011050613.16840: *5* UI notes
Both the legacy and new completer now work *exactly* the same way, because
they both use the AutoCompleterClass to compute the list of completions.

The strict "stateless" requirement means that the "intermediate"
completions must be entered into the body pane while completion is active.
It works well as a visual cue when using the tabbed completer: indeed, the
tabbed completer would be difficult to use without this cue.

The situation is slightly different with the qcompleter. Adding code before
the user accepts the completion might be considered an "advanced" feature.
However, it does have two important advantages, especially when "chaining"
across periods: it indicates the status of the chaining and it limits what
must appear in the qcompleter window.
.. @+node:ekr.20131011050613.16841: *5* Appearance
There is little change to the legacy completer, except that no text is
highlighted in the body pane during completion. This is calmer than before.
Furthermore, there is no longer any need for highlighting, because when the
user types a backspace the legacy completer now simply deletes a single
character instead of the highlighted text.

One minor change: the legacy completer now *does* insert characters that do
not match the start of any possible completion. This is an experimental
feature, but it might play well with using codewise completions as a
fallback to Leo-related completions.

.. @+node:ekr.20131011050613.16842: *5* Performance
Performance of Leo-related completions is *much* better than before. The
old code used Python's inspect module and was horribly complex. The new
code uses eval and is perfectly straightforward.

The present codewise-related code caches completions for all
previously-seen prefixes. This dramatically speeds up backspacing. Global
caching is possible because completions depend *only* on the present
prefix, *not* on the presently selected node. If ContextSniffer were used,
completions would depend on the selected node and caching would likely be
impractical. Despite these improvements, the performance of
codewise-oriented completions is noticeably slower than Leo-related
completions.

The ac.get_cached_options cuts back the prefix until it finds a cached
prefix. ac.compute_completion_list then uses this
(perhaps-way-too-long-list) as a starting point, and computes the final
completion list by calling g.itemsMatchingPrefixInList.

This may not be absolutely the fastest way, but it is much simpler and more
robust than attempting to do "prefix AI" based on comparing old and new
prefixes. Furthermore, this scheme is completely independent of the how
completions are actually computed. The autocompleter now caches options
lists, regardless of whether using eval or codewise.

In most cases the scheme is extremely fast: calls to get_completions
replace calls to g.itemsMatchingPrefixInList. However, for short prefixes,
the list that g.g.itemsMatchingPrefixInList scans can have thousands of
items. Scanning large lists can't be helped in any case for short prefixes.

Happily, the new scheme is still *completely* stateless: the completionDict
does *not* define state (it is valid everywhere) and no state variables had
to be added. In short, the new caching scheme is much better than before,
and it probably is close to optimal in most situations.
.. @+node:ekr.20131011050613.16843: *4* About Key handling
@language rest

The following sections deal with different aspects of how Leo handle's
keystrokes that the user types. This is the most complex code in Leo.
.. @+node:ekr.20131011050613.16849: *5* Key bindings
There are two kinds of bindings, gui bindings and pane bindings.

**Gui bindings** are the actual binding as seen by  whatever gui is in effect.
Leo binds every key that has a binding to k.masterKeyHander.

At present Leo makes gui bindings in several places, all equivalent.
Bindings are made to callbacks, all of which have this form::

     def callback(event=None,k=k,stroke=stroke):
        return k.masterKeyHandler(event,stroke)

As a result, changing gui bindings actually has no effect whatever.
It would be clearer to have a single place to make these bindings...

In any case, the purpose of these callbacks is to capture the value of 'stroke' so
that it can be passed to k.masterKeyHandler.
This relieves k.masterKeyHandler of the impossible task of computing the stroke from the event.

**Important**:  No function argument is ever passed to k.masterKeyHandler from these callbacks,
because k.masterKeyHandler binds keys to command handlers as described next.

**Pane bindings** are bindings represented by various Python dictionaries in the
keyHandlerClass (see below). k.masterKeyHandler and its helpers use these
dictionaries to call the proper command or mode handler. This logic is hairy,
but it is completely separate from the gui binding logic.

Here are the dictionaries that k.masterKeyHandler uses:

- c.commandsDict:
  Keys are minibuffer command names; values are functions f.

- k.inverseCommandsDict:
  Keys are f.__name__l values are emacs command names.

- k.bindingsDict:
  Keys are shortcuts; values are *lists* of g.bunch(func,name,warningGiven).

- k.masterBindingsDict:
  Keys are pane names: 'all','text',etc. or mode names.
  Values are dicts:  keys are strokes; values are g.Bunch(commandName,func,pane,stroke).

- k.modeWidgetsDict:
  Keys are mode names; values are lists of widgets to which bindings have been made.

- k.settingsNameDict:
  Keys are lowercase settings; values are 'real' Tk key specifiers.
  Important: this table has no inverse.

- inverseBindingDict:
  This is *not* an ivar; it is computed by k.computeInverseBindingDict().
  Keys are emacs command names; values are *lists* of shortcuts.
.. @+node:ekr.20131011050613.16850: *5* Handling key events
.. @+node:ekr.20131011050613.16851: *6* All events are key events
All event objects passed around Leo are *key* event objects. Taking a look
at the eventFilter method, we see clearly see that *only* key events ever
get passed to Leo's core. All other events are handled by Qt-specific event
handlers.

As can be seen, these non-key events *can* be passed to Leo, but only as
the event arg in g.doHook (!) At present, no plugin ever calls
k.masterKeyHandler. The only call to k.masterKeyHandler in qtGui.py is the
expected call in eventFilter.

There are other calls to k.masterKeyHandler in Leo's core, but we can prove
(by induction, if you will), that all events passed to k.masterKeyHandler
are proper leoKeyEvent objects.
.. @+node:ekr.20131011050613.16852: *6* c.check_event
The essential invariant is that the events passed to Leo's core methods
really are leoKeyEvent objects created by qtGui.py. Rather than asserting
this invariant, the code will contains calls to c.check_event in essential
places. c.check_event is a "relaxed" place to do as much error checking is
needed. In particular, running the unit tests calls c.check_event many
times.

c.check_event is a happy "accident". It turns out to be the essential
consistency check that continually verifies that the Qt event methods are
delivering the expected keys to k.masterKeyHandler.
.. @+node:ekr.20131011050613.16853: *5* About the KeyStroke class
The KeyStroke class distinguish between "raw" user settings
and the "canonicalized" form used throughout Leo. Indeed,
the ability to explicitly distinguish between the two, using
type checking, substantially clarifies and simplifies the
code.

The KeyStroke class makes possible vital type-related
assertions. Knowing *for sure* exactly what crucial data is
and what it means is a huge step forward.

Objects of the KeyStroke class can be used *exactly* as a
strings may be used:

A. KeyStroke objects may be used as dictionary keys, because
they have __hash__ methods and all the so-called rich
comparison methods: __eq__, __ne__, __ge__, __gt__, __le__
and __lt__. Note that KeyStroke objects may be compared with
other KeyStroke objects, strings and None.

B. At present, KeyStroke objects supports the find, lower
and startswith methods. This simplifies the code
substantially: we can apply these methods to either strings
or KeyStroke objects, so there is no need to create
different versions of the code depending on the value of
g.new_strokes.

However, having the KeyStroke class support string methods
is bad design. Indeed, it is a symptom that the client code
that uses KeyStroke objects knows too much about the
internals of KeyStroke objects. Instead, the KeyStroke class
should have higher-level methods that use s.find, s.lower
and s.startswith internally.

You could say that the fact that code in leoKeys.py calls
s.find, s.lower and s.startswith is a symptom of non OO
programming. The internal details of settings and strokes
"pollutes" the code. This must be fixed. This will likely
create opportunities for further simplifications.

> Why not just have .s attribute in KeyStroke, that contains
the string version?

It is truly impossible to understand the key code without
knowing whether an object is a string representing a user
setting or the canonicalized version used in Leo's core,
that is, a KeyStroke object. Using ks.s instead of ks
destroys precisely the information needed to understand the
code.

Again, this is not a theoretical concern. The key code now
contains assertions of the form::

    assert g.isStroke(stroke)

or::

    assert g.isStrokeOrNone(stroke)

Getting these assertions to pass in *all* situations
required several important revisions of the code. The code
that makes the assertions pass is "innocuous", that is,
almost invisible in the mass of code, but obviously, these
small pieces of code are vital.

This is in no way a violation of OO principles. The code is
not dispatching on the type of objects, it is merely
enforcing vital consistency checks. This code is complex:
confusion about the types of objects is intolerable.
Happily, the resulting clarity allows the code to be
substantially simpler than it would otherwise be, which in
turn clarifies the code further, and so on...
.. @+node:ekr.20131011050613.16854: *5* Simplifying the Qt input code
The Qt key input code can be greatly simplified by calling a
new k.makeKeyStrokeFromData factory method. At present, the
Qt key input code knows *all* the details of the format of
*canonicalized* settings. This is absolutely wretched
design.

Instead, the Qt input key code should simply pass the key
modifiers and other key information to
k.makeKeyStrokeFromData, in a some kind of "easy" format.
For example, the Qt input key code would represent the
internal Qt modifiers as lists of strings like "alt",
"ctrl", "meta", "shift". k.makeKeyStrokeFromData would then
create a *user* setting from the components, and then call
k.strokeFromSetting to complete the transformation.
.. @+node:ekr.20131011050613.16855: *5* Typed dicts
Leo's key dictionaries will always be complex, but basing
them on the TypedDict class was a major improvement.

The g.TypedDict and g.TypedDictOfLists classes are useful
for more than type checking: they have unique names and a
dump method that dumps the dict in an easy-to-read format
that includes the name, and valid types for keys and values.

Plain dicts do have their uses, but for "long-lived" dicts,
and dicts passed around between methods, plain dicts are as
ill-advised as g.Bunches.

.. @+node:ekr.20130807203905.16683: *4* ENB: fixing the key binding bug
From: "Edward K. Ream" <edreamleo@gmail.com>

As the title indicates, this thread will consist of what could be
called an online engineering notebook.  Please feel free to ignore.

Key bindings are one of the most difficult and complex parts of Leo.
This can't be helped: Leo's goals for key bindings are ambitious.

The present bug, https://bugs.launchpad.net/leo-editor/+bug/879331, is
due to a significant design oversight.  Redefining a binding for a
command x to key y affects not just command x but all other commands
presently bound to y!

Alas, the present binding tables are already complex.  I am almost at
the limit of what I can hold in working memory as it is.  Adding
significant additional complexity risks creating virtually impossible-
to-understand code.

There are two conflicting desires in play here:

1. To make the minimum changes needed.  While reasonable in itself,
this promises to increase overall complexity.

2. To decrease overall complexity.  While reasonable in itself, this
promises significant overall changes to the code.

Combining these two desires yields a strategy of finding a minimal
change that reduces overall complexity :-)  It's a big ask.
.. @+node:ekr.20031218072017.367: *4* How to add support for a new language
- Add new entries in the following Python dictionariues in leoApp.py:
  self.language_delims_dict, self.language_extension_dict and self.extension_dict

- Add an entry to the languages list in <<configure language-specific settings>>

- Add a list of the keywords of the language to << define colorizer keywords >>

  N.B.: the name of this list must be x_keywords, where x is the entry in language in step a.

- Add any language-specifig code to leoColor.colorizeAnyLanguage.
  For most languages nothing need be done.

- If the language is case insensitive, add it to the list of
case_insensitiveLanguages found in  << define global colorizer data >>

- Create the files theLanguage.xml and theLanguage.py file was added to the leo\modes directory.
  See Chapter 15 of Leo's users guide for full details.

TESTS

- Test the syntax coloring for the new language by using the @language directive.

@color
.. @+node:ekr.20070623150151: *4* How to make Leo commands undoable
The chapter commands provide a good example.  In this file, see the node:

Code-->Core classes...-->@thin leoChapters.py-->class chapterController-->Undo

The general plan is this:

1. The command handler calls a **beforeCommand** method before changing the outline.

The beforeCommand method creates a g.Bunch that contains all the information needed to
restore the outline to its previous state. Typically, the beforeCommand method
will call c.undoer.createCommonBunch(p), where p is, as usual, c.p.

2. After changing the outline the command handler calls an **afterCommand** method.

This method should take as one argument the g.Bunch returned by the
beforeCommand method. In the discussion below, denote this bunch by b. The
afterCommand method adds any information required to redo the operation after
the operation has been undone.

The afterCommand method also sets b.undoHelper and b.redoHelper to two method
that actually perform the undo and redo operations. (Actually, the beforeCommand
method could also set these two entries).

When the afterCommand method has 'filled in' all the entries of b, the
afterCommand method must call u.pushBead(b). This pushes all the undo
operation on a stack managed by the Leo's undoer, i.e., c.commands.undoer.

3. The undoer calls the undoHelper and redoHelper methods to perform the actual undo and redo operations.

The undoer handles most of the housekeeping chores related to undo and redo.  All the undoHelper and redoHelper methods have to do is actually alter Leo's outline.

**Note**: the undoer creates an ivar (instance variable) of the *undoer* class for every entry in the bunch b passed as an argument to u.pushBead(b).  For example, suppose u = c.commands.under and that b has ivars 'a','b' and 'c'.  Then, on entry to the undoHelper and the redoHelper the u.a, u.b and u.c ivars will be defined.  This makes it unnecessary for the undoHelper or the redoHelper to 'unpack' b explicitly.

Writing correct undo and redo helpers is usually a bit tricky.  The code is often subtly different from the original code that implements a command.  That just can't be helped.




.. @+node:ekr.20131017100502.16702: *4* How to profile Leo
To gather statistics, do the following in a console window, not idle::

    > python
    >>> import leo
    >>> import leo.core.runLeo as r
    >>> r.prof()  (this runs leo)
    load any .leo file from Leo
    quit Leo

This writes intermediate data to cwd.leoProfile.txt.
The statistics are written to stdout.
.. @+node:ekr.20050214055018: *4* Mac Notes
.. @+node:ekr.20050221054932: *5* How to make monolithic Leo app on MacOS X
 @killcolor
http://sourceforge.net/forum/message.php?msg_id=3007062
By: jgleeson

Sorry to take so long to reply.  I've been buried in work and haven't kept up
with some email.

Here's the link to the site where I posted the folder you have:
<http://homepage.mac.com/jdgleeson/>  It's the small file named "Leo.zip" (23
KB), not the large file "Leo-4.3-alpha-2.dmg" (20 MB).

I agree that I did not write very clear instructions, beginnng with the first
step, where I should have also said:  "It is important to use version 1.1.8
of py2app, which is only available through svn.  The version on the py2app website
is 1.1.7, which creates buggy Tkinter apps. If you try to use version 1.1.7,
the Leo app it creates will give you a message saying that Tkinter is not properly
installed.  Your installation is fine; otherwise you could not have even built
Leo.app with py2app, because py2app copies the essential parts of Tcl/Tk into
the application bundle to make the app completely standalone."

I haven't tried intalling the Fink subversion -- I'm using DarwinPorts
<http://darwinports.opendarwin.org/>.  But there's a simpler alternative than
DarwinPorts. Metissian releases OS X packages of Subversion clients
<http://metissian.com/projects/macosx/subversion/>

AFAIK, the command "python setup.py bdist_mpkg --open" only applies to the py2app
1.1.8 distribution.  By the way, bdist_mpkg is distributed with py2app. It creates
a package around the setup.py script (more specialized than Platypus).  I don't
have any experience with bdist_mpkg yet.

'Copy the leo folder into this directory' is horrible. I'm glad you figured
it out -- I'm not sure I could have.

"python setup.py py2app -a" should be run in the folder with the readme file,
which also contains the setup.py file that the command refers to.  Most importantly,
the folder in which this command is run must contain the leo folder -- which
it does only if you are brilliant enough to decode my instructions.   ;) 

HTH

-John
.. @+node:ekr.20050214055018.4: *5* @url http://idisk.mac.com/genthaler-Public/Leo.zip (download)
.. @+node:ekr.20050214055018.5: *5* @url http://www.wordtech-software.com/leo.html  (Mac Bundle)
.. @+node:ekr.20050513164506: *5* Problems with run script command on Mac x11
@killcolor

Jon Schull <jschull@softlock.com>  
Date:  2003/12/30 Tue PM 05:50:51 EST 
To:  edreamleo@charter.net 
Subject:  Leo, Mac OS X 10.3, and VPython 

I've been evaluating leo or vpython programming on  Mac OS X 10.3, and 
have some observations and a suggestion.

Observations:
- Leo runs under X11 as well as under OS X.
- My X11 python configuration was created using the recipe at XXX (which enables vpython).
- The OS X configuration is vanilla MacPython from MacPython.org, along with AquaTclTk batteries included XXX.

In both environments I can run leo under python leo.py and under idle.
Under OS X we get font smoothing, but we can't run visual python programs (python crashes;  this is a known incompatibility with  MacPython.)

- Under X11 we can run visual python programs like this one
    #box.py
    from visual import *
    box()

And we can even run them under leo (under X11). HOWEVER, when the visual python program is terminated, leo vanishes (leo and the vp program apparently run in the same space)

Under x11, we can keep leo alive by putting the vp program in its own space:

    os.popen3('/sw/bin/python /Users/jis/box.py')

However,  this doesn't let us see the output of stderr and stdout.  
Those text streams are available...

    def do(cmd='ls'):
        from os import popen3
        pIn,pOut,pErr=0,1,2
        popenResults=popen3(cmd)
        print popenResults[pOut].read()
        print popenResults[pErr].read()

    import os
    do('/sw/bin/python /Users/jis/box.py')

...but only when the vpython program terminates.

Here's the good news:  if we execute our vp program with 
/sw/bin/idle.py rather than with python, we get to see the program 
output in real time (under idle, under X11).

    import os
    os.chdir('/sw/lib/python2.3/idlelib')
    os.popen3('/sw/bin/python idle.py -r /Users/jis/box.py')

#this runs as an executed script in leo, and produces a live idle 
with real time ongoing output.

Now, while idle is running, leo sits in suspended animation.  But when 
the vpython program terminates, we are left in idle, and when idle is 
terminated, leo becomes active again.

It would be even better if leo were not suspended (using os.spawn, 
perhaps) but the real point is that I would really really like leo's 
"Execute script" command to execute code this way and spare me having 
to  hard-write the path to box.py.  It ought to be possible to 
eliminate os.chdir as well.

------------------
Jon Schull, Ph.D.
Associate Professor
Information Technology
Rochester Institute of Technology
schull@digitalgoods.com 585-738-6696
.. @+node:ekr.20040104162835.8: *5* Linux/Mac notes: Dan Winkler
.. @+node:ekr.20040104162835.13: *6* Fink & aqua
Yes, fink does have pre-built Pythons, both 2.1 and 2.2.  (If you don't 
see them it probably means you don't have the right servers listed in 
your /sw/etc/apt/sources.list file.)  However, the versions of Python 
you'd get through fink are set up to run under X Windows, which I don't 
think is what you want.

I think what you want is MacPython which can run Tk programs like Leo 
under Aqua.  That's what I use these days.

I can tell from your question that you don't understand the following 
differences between the versions of Python available:

1) The version that comes with OS X is a text only one which doesn't 
have Tk.  Leo can't run under that.  Also, I hate Apple for including 
this instead of one that does have Tk and I hope they'll fix it some 
day.

2) You can get a version of Python from fink with has Tk but which runs 
under X Windows.  I don't think you want that.

3). You can also get MacPython which has Tk but it's a version of Tk 
that uses the Aqua windowing system, not X Windows.

So Tk can either be present or not and if it is present it can use 
either X Windows or Aqua.  You want it present and using Aqua, I think.


.. @+node:ekr.20040104162835.14: *6* Mac, Fink, etc.
> 1. The python that FC installs is MacPython.  I think that because the
> MacPython docs talk about Fink.

Nope.  The python installed by FC knows nothing about the Mac.  It 
thinks it's running on a Unix machine.  And it uses a version of Tk 
which thinks it's running on a Unix machine.  The window standard on 
Unix is called X (or X11 or XFree86, all the same thing).  So the main 
reason to run Leo this way would be to get an idea of how it works for 
Unix/Linux users.  But when programs run under X, they don't look like 
Mac programs.  They don't get all those glossy, translucent widgets 
that Aqua provides.  They really look like they would on a Unix/Linux 
machine.

Aqua is the native windowing system on Mac.  MacPython is set up to 
work with it.  Most Mac users will want Leo to work this way.  That's 
what I do.

>
>
> I have the TkTclAquBI (Batteries included) installer.  Is installing 
> this
> enough to get Leo to work with Aqua?  Do I have to de-install the
> present tk stuff that I installed with FC?

Yes, I think that's all I installed to get Tk to work under Aqua.  You 
don't have to deinstall the FC stuff.  All the FC stuff lives in its 
own world under /sw and runs under X.  It won't conflict with the Mac 
world.

.. @+node:ekr.20040104162835.15: *6* Double clicking on Linux
Double-clickable things (i.e. Macintosh applications) are usually 
actually folders with a name that ends in .app.  The file you found is 
probably executable only from the command line, not by double clicking 
it.  So I think if you run it from the command line it will work but 
will not know about Tk because Apple's version was built without Tk 
support.

You can also execute the .app programs from the command line by using 
the open command, so "open foo.app" will do the same thing as double 
clicking on foo in the finder (the .app extension is suppressed).  The 
idea behind this is that an application can look like just one opaque 
icon in the finder but actually have all its resources nicely organized 
in subfolders.
.. @+node:ekr.20110527084258.18374: *4* Summary of the Ashland sprint
http://groups.google.com/group/leo-editor/browse_thread/thread/9b1dbebd56d50e14/d5a690127ddad38e
 
"Sprint" isn't really the correct term.  We wrote no code.  Instead,
we discussed what seemed to each of us as the most important
directions for Leo.

After much pleasant discussion, we reached agreement, dreaded or not,
on just about everything. I'll summarize the topics here, and
elaborate about file format issues in a separate thread.

1.  (Done) Make .leo files as standards-compliant as possible.

This will demonstrate to newbies that we have some sophistication re
web standards, and it will allow external tools to handle .leo files
in the easiest possible way.

I've delegated the design of this project to Kent.  I'll be in charge
of implementation.

2. Simplify and revise Leo's file format.

Details in a separate thread: some items deserve just a bit more
discussion.  In particular, we want the about-to-be-renamed <t>
elements to contain headline text so that <t> elements represent key/
value pairs directly.  The question is, should the about-to-be-renamed
<v> elements contain headline text (readable, but "denormalized")?

3. Support reading and writing Leo outlines in JSON format.

This will allow closer cooperation with databases and other tools.
I'll do this.

4. Complete the transition to Terry's free_layout plugin.

A. Place separate body editors in free_layout areas.  This should
*easy* to do!  Almost nothing changes in the code, but the visual
effect should be much better.

B. Allow any pane to be "tabified" (placed in a tab in the Log pane)
and "untabified."  There are a few details to be handled, but nothing
major.

Terry and I will collaborate on this.

5. Add global search to the quicksearch plugin and to Leo's find
command.

6. (Abandoned) Add node-specific undo asap.

The present undo is almost useless after a few levels.  Node-specific
undo would be much more useful. This has been on the list forever.  It
should be done yesterday.

7.  Rejected direct support for .ini files instead of Leo's @settings
nodes.

After some discussion we decided that the present .ini importer should
suffice.  In other words, it seems like a bad idea to support .ini
settings *instead* of, or in *addition* to, .ini files.  However,
scripts or commands to import/export Leo settings to one or more .ini
files would be fine.

8.  (done) Make uA's first class citizens.

There should be commands to get and set uA's.  This is easy to do:
it's just an oversight.

Summary
=======

Looking back on the discussions, I am struck once again by how minor
the suggestions are.  Most of these items can be done in a day or two,
or a week or two at most.  The conclusion is that Leo has reached a
mature state.

Kent, Terry, did I omit anything?  Misstate anything? 
.. @+node:ekr.20140212062130.16585: *4* Theory of operation for @auto-view
@language rest


.. @+node:ekr.20140212062130.16586: *5* Distinction: bare and nested organizers
Two major new Aha's have appeared. As often happens, they were the result
of a new distinction:

  **Nested** organizer nodes will become children of *other* organizer nodes.

  **Bare** organizer nodes will become children of *ordinary* nodes.
.. @+node:ekr.20140212062130.16587: *5* Two Ahas re nested organizers
Aha 1: We don't have to compute child indices for nodes moved to children
of organizer nodes!

We simply put them on a list (in imported outline order) of tuples
(parent,moved_node). There will be only *one* such list, say
vc.global_moved_node_list.

When the time comes to do the actual moves, we do the following:

- Reverse the global list, thereby guaranteeing that positions that appear
   later in the reversed list remain valid.
- Insert each item on the list as the *last* child of the parent
   (organizer) node.
   
​There is a small, easily corrected, hole in this argument. It assumes that
positions appear in the list in outline order. That's usually so, but not
always. Indeed, @organizer: nodes *are* written in outline order, as are
their contents, but rearrangements in the imported outline could
conceivably mean that the nodes corresponding to @organizer nodes would not
be handled in the *new* outline order. This is easily remedied: just sort
the list in (imported) outline order before reversing it

That's all!

Using a global list was hard to see, because typically such schemes don't
work in recursive environments. But the @auto-view algorithms are *not*
recursive. Each @organizer: node is handled separately, in a completely
linear fashion, by vc.demote_helper, the true main line of the code.

Aha 2:  We can treat *nested* organizer nodes just as we treat ordinary nodes!

vc.switch_active_organizer (the code that handles the entry into an
organizer node) will simply add another entry (parent,organizer_node) to
the global move list.

Not having to track most child indices is a major collapse in complexity.
Indeed, the code that moves nodes to their final destinations will have no
"if" statements at all. It doesn't get any more solid than that.
.. @+node:ekr.20140212062130.16588: *5* Moving bare organizers
===== Moving bare organizer nodes.

Now that moving most nodes has become trivial, the only remaining task is
to *safely* move bare organizers. This task has two parts:

1. Determine the proper child index of *bare* organizer nodes.

This will be done in the main line: vc.demote_helper will maintain a count
n of imported nodes that *haven't* been assigned to organizer nodes. This
count must also be incremented to include bare organizer nodes the will be
inserted later. It's just a bit tricky, but there are only a few cases to
get right. Once it works, it will work forever.

There will be another global list, say vc.global_bare_organizer_node_list,
containing tuples (parent,bare_organizer,n). vc.switch_active_organizer
will add entries to this list, but *only* for bare organizer nodes.

2. Actually move the bare organizer nodes.

The children of bare organizer nodes will appear vc.global_moved_node_list.
Each (ordinary) parent node may contain several bare organizer nodes. Here
is how to get the job done safely:

- For each parent, create a new per-parent list, containing all entries of
  the global list with that parent.
- Sort each per-parent list using n as the key.
- Move the nodes in each per-parent list to their destinations, in sorted
  order.

At first glace, this seems entirely bogus. How are we to ensure that
positions stay valid?

Unlike for ordinary nodes, we can't process bare organizer nodes in reverse
outline order: we have to handle them in the *new* sibling order, that is,
sorted by n. In other words, it won't work to move them to the last child
of their parent! Nor can we move nodes to any arbitrary sibling position:
we must move nodes with smaller child indices before moving nodes with
larger child indices.

To solve this problem, the setup code places each organizer node in a
separate **holding cell**. Moving the bare organizer out of a holding cell
to its final resting place affects no other holding cells, and no other
organizer node.


.. @+node:ekr.20140212062130.16589: *5* Intermediate organizers
This post contains important insights for both users and developers.

The symptom of the bugs is that *non-bare* organizer nodes are not placed
in correct sibling order. As you recall, the value of n in the main line,
demote_helper, matters only for *bare* organizer nodes. So the proximate
cause of the problem is that pending nodes aren't entered on the global
node list before entering the organizers.

As we shall see below, the more fundamental problem is that the code does
not fully account for the outline relationships between organizer nodes. As
a result, the existing code does not always open and close organizer nodes
properly. (An organizer node is open when it may accumulated organized
nodes; an organizer node is closed when it can no longer accumulate
organized nodes without changing the order of the nodes in the @auto file.)

Two examples will illustrate the subtleties, and will lead naturally to a
new distinction, that of an **intermediate organizer** node.

Let's start with an easy case.  Suppose the organizer structure is:

  - @auto x.py
    - leading nodes
    - organizer A
       - node 1
       - organizer B
           - node 2
       - organizer C
       - node 3
    - organizer D

Clearly, the children of organizer A should be: node1, organizer B,
organizer C and node 3. That is, we must *not* close organizer A just
because we have entered organizer B or organizer C. Otoh, we *must* close
organizer A when we see organizer D.

The rule is clear: we must close an organizer whenever we see another
organizer that is not a descendant organizer.

The existing code creates parent/child data for OrganizerData instances. It
will be straightforward to add descendant relationships.

Let's consider another case:

- @auto x.py
    - leading nodes
    - organizer A
       - organizer B
         - organizer C
           - node 1
 
Here, organizer B is an **intermediate organizer** node. That is, it occurs
"between" organizers A and C.

Adding node 1 to the global node list should open organizer B, but the
present code does not. The present code opens an organizer node only when
adding a node to the organizer, but this example shows that organizer B
must be added to the global node list *before* any node is added to it.
Failure to do so results in organizer B being put in the wrong place.
.. @+node:ekr.20140212062130.16590: *5* Aha: organizers relationships are stable
As I was thinking about the outline relationships between organizer nodes,
I began to worry that minor changes to the imported @auto file might break
those relationships. In fact, the reverse is true: the relationships
between organizer nodes are *independent* of the @auto file!

The reason is simple: those relationships are created by the unl's in
@organizer: nodes in the .leo file. Furthermore, reorganizing the *outline*
will automatically update the unls in the @organizer nodes (in the @views
node) when the outline is saved. Thus, the *relationships* between
organizer nodes can never be broken by changes to the @auto node! This is
really important for users to know.

And there is more good news. Changing the contents of the @auto files can
indeed change the contents of organizer nodes, but the only way to totally
invalidate an organizer node is to change the *structural* relationships of
data in the @auto file. This almost never happens!

For example, if aClass is defined at the top level of the @auto file, then
the only way to invalidate the organizers that contain aClass or its
members is to move aClass so it is not a top-level node. But Leo's
importers will put aClass at the top level unless it becomes a nested class
in some other class! To repeat, this kind massive change to the context of
a class, function or method almost never happens.

In short, the links between unls (in @organizer: nodes) and imported
classes, methods and functions almost never break! This is truly great
news.
.. @+node:ekr.20140212062130.16591: *5* Summary
The following insights are important for users (and maintainers) to know:

- Changing an @auto file does not change the relationships between its
  organizer nodes in any way.
- Saving an outline automatically updates the relationships between
  organizer nodes.
- Unl's are much more stable than one might suppose: the *only* way to
  "break" organizer nodes is to change the fundamental relationships of
  classes, methods or functions in the imported @auto file. This almost
  *never* happens.
.. @+node:ekr.20140212062130.16592: *5* Headines are important
Headlines matter The unls will break if the headlines produced by *the
importer* don't match those unls. There are two cases to consider:

A. Section references. At present, Leo will abort the writing of an @auto
    tree that contains section references. Instead, Leo could write the
    expansion of the section reference. In practice, changing section
    definition nodes to ordinary nodes by hand isn't a big deal, but it
    would be nice for Leo to do that for the user.

B. Headlines for classes, methods and functions.

This is the important case. Most of my headlines look like x.y (z), where x
is some kind of indication of the class and y is the actual class, method
or function name and z is a optional comment. When converting from @file to
@auto *all* of the organizers will break unless the headline becomes just
"y".

Perhaps the best solution is to ask the importer what name it will give to
the node. Yes, this involves parsing the contents of the node. It will
likely require a new interface method of the baseScannerClass.
.. @+node:ekr.20140212062130.16593: *5* Existing organizer nodes
There are *two* kinds of organizer nodes:

- those that do *not* exist in the imported outline (these now work).
- those that *do* exist in the imported outline.

For example, I usually put the helpers of method X as children of X. There
is *no way* for the read logic to recreate such nodes at present, because
the parent/child relationships are do not exist in the @auto file. So X is
an organizer node!

Happily, it should be relatively straightforward to support **existing
organizers**:

1. The write code will represent existing organizers with
    @existing-organizer: nodes rather than @organizer: nodes. As with
    @organizer: nodes, the headline will contain the headline of the
    organizer node; the body will be a list of unls.

The write code can easily discover existing organizers: they are nodes with
children that contain something other than comments and blank lines.

2. The read code will create OrganizerData entries for @existing-organizer
   nodes just as with @organizer: nodes. A new ivar, data.existing, will
   distinguish between @organizer: and @existing-organizer nodes. These new
   entries will be members of a new global list of all organizer nodes.

The goal will be to confine tests of data.existing to the setup code. That
way, all the hard code will continue to work as is. We shall see whether
this is possible.

3. The code that munges unls will drop unls for a data object only if
   data.existing is False. This may be more tricky than it sounds. In any
   case, this is the fundamental reason why @existing-organizer: nodes
   *must* exist.
.. @+node:ekr.20140212062130.16594: *5* Fixing move_nodes_to_organizers
@language rest

After several hours staring at traces, I realized that *all* the problems
were in vc.move_nodes_to_organizers. That is, vc.global_moved_node_list,
the global move node list, was completely correct! This was not obvious,
because that trace in vc.move_nodes_to_organizers reported the *sorted*
version of the global move list. Bad idea: it was the sorting itself that
was causing all the problems!

Recall that I thought sorting the list was required to keep positions
unchanged until they are moved. But an hour or so of experimentation (at
3am) convinced me that there is *no way* to move items from sorted lists
into the correct positions.

After a few minutes of noodling, the solution finally appeared. Rather than
*moving* nodes by moving positions, the *only* way that leaves
to-be-considered positions intact is to *copy* all the nodes of a
position's tree to the correct place. Eureka! A new method,
vc.copy_tree_to_last_child_of does the actual copying. A final phase in
vc.move_nodes_to_organizers deletes all positions in reverse outline order.
So yes, sorting *is* essential, but only at the very end.

Note: these difficulties do not apply to bare organizer nodes, because each
is placed in its own holding cell. But it's not possible to create holding
cells for non-bare organizers.

.. @+node:ekr.20140212062130.16595: *5* Supporting @existing-organizers
Recall that *existing* organizer nodes are organizer nodes that the
importer will have created. Last night, after everything worked, I
reconsidered the design of existing organizer nodes in the light of the new
code. I wondered whether the existing code would correctly move such nodes
into their correct final positions.

The short answer is yes, because the importer will *not* have created any
children for existing organizer nodes. That being so, everything is
*exactly* as it is for organizer nodes, except that demote_helper and its
helpers never need to create the organizer. Some complications may arise,
but I expect them all to be minor.
.. @+node:ekr.20140212062130.16596: *5* Supporting custom headlines (@headlines)
In a previous post I wrote the new @auto write code could just warn if a
headline has changed since the @auto file was read. Wrong! It would be
*unbearable* to have to lose custom headlines. The new code must save and
restore headlines that don't match the **standard headlines**, the
headlines that the importer will give the node the *next* time the importer
reads the @auto file.

A new @headlines node (a child of the @auto-view node for the @auto file)
will contain the required data. It will contain pairs of lines, much like
@clones::

    unl: <a unl>
    head: <the custom headline>

The unls in the @headlines node must use standard headlines! The "links"
created by @headlines node will break if the body text of a node changes so
as to change its standard headline. That is, @headlines links will break
when class, method or function names change.

To combat such breakage, whenever the body text of a node changes the code
that writes the @headlines node should *recompute* the unl using a
to-be-written convenience method in leoImport.py. The convenience method
will return the new standard headline.
.. @+node:ekr.20140212062130.16597: *5* Automatic conversion of @file to @auto
When converting an @file node to @auto *every* body text must be considered
changed. All unls in the @headlines node must be set using the headline
that the importer will create for the corresponding node.
.. @+node:ekr.20140212062130.16598: *5* Why the new code is solid
​1. The hard part of the code involves moving nodes.

vc.global_moved_node_list is the backbone of the entire algorithm.
vc.demote_helper and its helpers add items to the list. vc.move_nodes and
its helpers remove items from the list. Very few significant "if"
statements exist:

Both vc.demote_helper and vc.move_nodes_to_organizers (the hard part of
vc.move_nodes) are straightforward. Very few if statements exist:

- The "main switch" in demote_helper is likely already correct and will
  have only localized consequences

- The single "if" statement in vc.move_nodes_to_organizers (the hard part
  of vc.move_nodes) reflects the fundamental distinction: nodes that must
  be copied vs. nodes that can be moved immediately.

2. Newly-installed traces make it clear what is happening.

This is surprisingly important. If bugs exist, it will be easy to recreate
them using unit tests that show, in compact, easy-to-understand dumps,
exactly what is happening.

3. Everything is easy after nodes have been moved.

Just for example, remembering nodes initial (standard) headlines is easy:
just create a dict whose keys are *vnodes* and whose values are the
standard headlines. A node's vnode does not change when the node moves, so
complications in the node-moving algorithm have *no* effect.

In short, only the node-moving algorithm (on both "sides" of the global
move list) has any potential for nasty bugs. The vast majority of those
bugs are gone, and will be easily fixed if any remain. Everything else is
completely routine bookkeeping.

4. Minimizing "if" statements.

I've worked hard to make all code as simple as possible.
.. @+node:ekr.20140212062130.16599: *5* docstring for vc.demote (not necessarily correct)
@language rest

'''
The main line of the @auto-view algorithm. Traverse root's entire tree,
placing items on the global work list.

About the algorithm:

Moving *any* kind of node to an *ordinary* (non-existing) organizer is
easy: each node is moved to the last child of the organizer.

Moving nodes to *existing* organizers requires exact housekeeping. This
code must know at what child index to insert a node.

Terminology:
- p is the to-be-moved node.
- a(p) is p's **anchor**, an existing node.  a(p) == p.parent(). 
- o(p) be the *existing* organizer to which p will be moved as a child.  
- d is the global **offset dict**, vc.anchor_offset_d.

There are two cases to consider when moving a node p to an existing node:

Existing organizer case 1: p is *any kind* of existing node (organizer or not).

We insert p as the n'th child of o, where:

    n = p.childIndex() + d.get(p.parent(),0)

Existing organizer case 2: p is an non-existing node.

p must be an organizer node (because it doesn't exist) and it must be a
*bare* organizer node because o(p) is an existing organizer.

Let p2 be the (existing) node that causes p to be inserted.

We insert p as the n'th child of o(p), where:

    n = p2.childIndex() + d.get(p2.parent(),0)

The only remaining part of the puzzle is calculating the offsets, that is,
the entries in d. The simplifying insights:

Insight 1: We only calculate offsets for *existing* nodes.

Insight 2: Because the organizer exists in this case,
           the anchor is the existing organizer: a(p) == o(p)

Insight 3. Only inserting or deleting nodes from existing node can change offsets.

With these insights firmly in mind we can consider the following offset
cases:

Offset case 1: visiting existing children of existing organizers.

By definition, the importer will create children of existing organizers.
These nodes must *not* be moved: they are already in their correct spots!

Offset case 2: adding a non-existing organizer node p to any existing (organizer) node o:

This case adds one to d.get(o)

Offset case 3: moving an existing node p to *any* organizer node, existing or not.

This case subtracts one from d.get(p.parent())
'''

# Note: o may be:
# - a bare (ordinary) organizer node,
# - an ordinary organizer node,
# - a bare existing organizer node
# - an existing organizer node that is a child of another organizer node.
.. @+node:ekr.20091218120633.6300: *3* Other notes
.. @+node:ekr.20141125112845.6: *4* jyLeo stuff
.. @+node:ekr.20051203084725.1: *5* How to install and run jythonShell
Install:

Put JythonShellEA.jar in c:\prog\JythonShell

(optional) Expand the jar so you can see the code:

jar xvf JythonShellEA.jar

Run:

Here is the contents of jythonShell.bat:

cd c:\prog\jythonShell
java -cp c:\jython-2.2a1\jython.jar;c:\prog\jythonShell\JythonShellEA2.1.jar org.leo.shell.JythonShell
.. @+node:ekr.20050316092232: *5* How to install jyLeo
- Unpack the .zip file, placing the result somewhere, say in c:\prog\jyleo-Jan-11-06

- Edit jleo.bat so it refers to jyleo-Jan-11-06.  For example:

rem open jyLeo
set ARGS= 
:loop 
if [%1] == [] goto end 
set ARGS=%ARGS% %1 
shift 
goto loop 
:end 

cd c:\prog\jyleo-Jan-11-06
java -jar c:\jython-2.2a1\jython.jar src\leo.py
.. @+node:ekr.20050716104357: *6* Old instructions
@language rest
- put the jyleo-nnn.jar file in c:\prog

- Execute the following command in a console window
    cd c:\prog
    jar xvf j-leo-nnn.jar

This creates a folder called j-leo-nnn

- Do the following, or execute jleo.bat

cd c:\prog\j-leo-nnn\src
java -jar c:\jython22a0\jython.jar leo.py

Note:  at present this gives KeyError: HOME
@language python

In leo.py, in computeHomeDir, I changed::

    home = os.getenv('HOME' )#,default=dotDir)

to::
    
    try:
        home = os.getenv('HOME' )#,default=dotDir)
    except Exception:
        home = ''

.. @+node:ekr.20050317153447: *6* jy-Leo install instructions by Paul Paterson
http://sourceforge.net/forum/message.php?msg_id=3053534
By: paulpaterson

Very interesting indeed - great work! 

I didn't have Java/Jython installed so for others in the same boat here's
what I had to do to get it work on my platform (Win2k). Some of this is in
the README but I had to do some extra but I'm not sure why.

1. Install 1.5 JDK  
http://java.sun.com/j2se/1.5.0/download.jsp 

2. Install Jython 
http://www.jython.org/jython22a1.zip 

3. Edit Jython.bat file - the part that calls Java.exe to ... 
"C:\Program Files\Java\jdk1.5.0_02\jre\bin\java" -cp "C:\Program Files\Java\jdk1.5.0_02\jre\lib";"c:\Apps\Python23\Jython";"C:\Apps\jLeo\j-leo-MAR15\Icons";"C:\Apps\jLeo\j-leo-MAR15\skins";"C:\Apps\jLeo\j-leo-MAR15\src";"C:\Apps\jLeo\j-leo-MAR15\skinimages" -Dpython.home="c:\Apps\Python23\Jython" -jar jython.jar %ARGS% 

Where  
- Java installed at C:\Program Files\Java\jdk1.5.0_02 
- Jython at c:\Apps\Python23\Jython 
- jLeo at C:\Apps\jLeo\j-leo-MAR15 

Change your paths as appropriate! There must be a better way to do this - Java confuses me! 

4. Edit leo.py in jleo/src directory to fix failure to find HOME env variable. 

line 241 becomes ... 

....try:home = os.getenv('HOME' )#,default=dotDir) 
....except KeyError:home="" 


Then, from the Jython install directory ... 

Jython " 
C:\Apps\jLeo\j-leo-MAR15\src\leo.py" 

Works a treat!  

Paul
.. @+node:ekr.20051129084430: *5* How to install jython
- Download jython_Release_2_2alpha1.jar and put it anywhere (say on the desktop)

- Double-click the file.  This brings up an installer.  Follow the direction.
  (I installed to c:\jython-2.2a1

- Using the Control Panel, System, Advanced tab, environment variables,
  add c:\jython-2.2a1\jython.jar to CLASSPATH (in user variables)
.. @+node:ekr.20051129084430.1: *6* @url http://www.jython.org/install.html
.. @+node:ekr.20060111112513.1: *5* New jyLeo notes
http://sourceforge.net/forum/message.php?msg_id=3516227
By: leouser

Some highlights:
* simpler startup:
jyleo leo.py
should be sufficient to start it up.
* new editor colorization
* the JythonShell is much more powerful and cooler
* new plugins
* Chapters support
* mod_script is in place.
* dyna-menu was converted.  I guess 'e' will have to judge the conversion.
* multi-language script support.
* drag and drop
* some powerful new editor commands.  Try keyword completing on the language
in effect.  Say if it is python:
se(Tab)
becomes
self

Some warnings:
1. Be careful about reading your regular leo files into jyleo and saving them.
Its quite conceivable that jyleo will write it out to an XML format that regular
leo can't handle.  Why?  Well jyleo is using an XML library to spit its XML
out while leo uses a home grown method.  The library can handle leo's XML, but
Ive seen regular leo not be able to handle jyleo's XML.  Its based around <tag/>
I believe.

2. If you move jyleo after executing it you will need to clear out your compiled
py files as the __file__ attribute is hard compiled into the resulting objects.
Not what we want.  We want it to be set at runtime.  Ive been waiting a long
time for jython to release again and hopefully fix this, but Im not holding
my breath anymore.

----------
Its hard to give this thing a number, I want to call it jyleo2, but jyleo is
sufficient.  Dependent upon bug reports the next release could be much sooner
than before, maybe even weeks.  I hope one thing, that the dreaded "I can't
get it to start" problems are gone.  I took the snapshot and expanded it in
Windows XP.  Went to the src directory and typed: jython leo.py
and it started.  That's what I wanted to see.  I didn't have to mess with the
CLASSPATH or anything.

things needed:
java 5
a jython2.2a1 or beyond.  jython2.2a1 is the most recent snapshot.

Beyond bug fixing, I will be planning to add more SwingMacs command as time
goes along.  But I think most major features are in place.  Of course the 3D
experiments in the future could change that... :D

A NOTE ON STARTUP TIMES: In my experience it takes awhile for jyleo to start.
It will take much longer the first time you execute it because the py files
are being compiled.  Ive haven't been able to figure out what eats the time,
it may just have a slow startup in the aggregate.  So don't think its not doing
anything, it probably is.
.. @+node:ekr.20071104222805: *4* Emacs/Pymacs notes
.. @+node:ekr.20071102191642.1: *5* xemacs/pymacs install notes
Added the following line to setup.py and setup files.

# -*- coding: utf-8 -*-

The second installation script installs the Emacs Lisp part only.
[snip]
I couldn't get this script to work.  Instead, I just created a pymacs folder at::

    C:\XEmacs\xemacs-packages\lisp\pymacs

For Win32 systems, I created create c:\Windows\pymacs-services.bat containing::

    c:\Python25\python C:\prog\Pymacs-0.22\scripts\pymacs-services

To check that pymacs.el is properly installed, start Emacs and do::

    M-x load-library RET pymacs

You should not receive any error.
(works)

To check that pymacs.py is properly installed, start an interactive Python session and type::

    from Pymacs import lisp

you should not receive any error.
(works)

To check that pymacs-services is properly installed, type the following in a console::

    pymacs-services </dev/null

You should then get a line ending with (pymacs-version version), and another saying : Protocol error : `>' expected..
(works, mostly: I omitted the </dev/null

The rest is from Leo's Chapter 18::

    ; Step 1: load leoPymacs if it has not already been loaded.
    (setq reload nil)
    (if (or reload (not (boundp 'leoPymacs)))
        (setq leoPymacs (pymacs-load "leoPymacs" "leo-"))
        (message "leoPymacs already loaded")
    )

    ; Step 2: compute the path to leo/test/ut.leo using a Leo script.
    (setq script
        "g.app.scriptResult = g.os_path_abspath(
            g.os_path_join(g.app.loadDir,'..','test','ut.leo'))"
    )
    (setq fileName (leo-run-script nil script))

    ; Step 3: execute a script in ut.leo.
    (setq c (leo-open fileName))
    (setq script "print 'c',c.shortFileName() ,'current:',c.p.h")
    (leo-run-script c script)
.. @+node:ekr.20071103090504: *5* Pymacs docs
@killcolor
.. @+node:ekr.20071103090504.1: *6* Emacs Lisp structures and Python objects
.. @+node:ekr.20071103090504.2: *7* Emacs lisp structures and Python
Conversions

Whenever Emacs Lisp calls Python functions giving them arguments, these arguments are Emacs Lisp structures that should be converted into Python objects in some way. Conversely, whenever Python calls Emacs Lisp functions, the arguments are Python objects that should be received as Emacs Lisp structures. We need some conventions for doing such conversions.

Conversions generally transmit mutable Emacs Lisp structures as mutable objects on the Python side, in such a way that transforming the object in Python will effectively transform the structure on the Emacs Lisp side (strings are handled a bit specially however, see below). The other way around, Python objects transmitted to Emacs Lisp often loose their mutability, so transforming the Emacs Lisp structure is not reflected on the Python side.

Pymacs sticks to standard Emacs Lisp, it explicitly avoids various Emacs Lisp extensions. One goal for many Pymacs users is taking some distance from Emacs Lisp, so Pymacs is not overly pushing users deeper into it.
.. @+node:ekr.20071103090504.3: *7* Simple objects
Simple objects

Emacs Lisp nil and the equivalent Emacs Lisp () yield Python None. Python None and the Python empty list [] are returned as nil in Emacs Lisp.

Emacs Lisp numbers, either integer or floating, are converted in equivalent Python numbers. Emacs Lisp characters are really numbers and yield Python numbers. In the other direction, Python numbers are converted into Emacs Lisp numbers, with the exception of long Python integers and complex numbers.

Emacs Lisp strings are usually converted into equivalent Python narrow strings. As Python strings do not have text properties, these are not reflected. This may be changed by setting the pymacs-mutable-strings option : if this variable is not nil, Emacs Lisp strings are then transmitted opaquely. Python strings, except Unicode, are always converted into Emacs Lisp strings.

Emacs Lisp symbols yield the special lisp.symbol or lisp[string] notations on the Python side. The first notation is used when the Emacs Lisp symbol starts with a letter, and contains only letters, digits and hyphens, in which case Emacs Lisp hyphens get replaced by Python underscores. This convention is welcome, as Emacs Lisp programmers commonly prefer using dashes, where Python programmers use underlines. Otherwise, the second notation is used. Conversely, lisp.symbol on the Python side yields an Emacs Lisp symbol with underscores replaced with hyphens, while lisp[string] corresponds to an Emacs Lisp symbol printed with that string which, of course, should then be a valid Emacs Lisp symbol name.
.. @+node:ekr.20071103090504.4: *7* Sequences
Sequences

The case of strings has been discussed in the previous section.

Proper Emacs Lisp lists, those for which the cdr of last cell is nil, are normally transmitted opaquely to Python. If pymacs-forget-mutability is set, or if Python later asks for these to be expanded, proper Emacs Lisp lists get converted into Python lists, if we except the empty list, which is always converted as Python None. In the other direction, Python lists are always converted into proper Emacs Lisp lists.

Emacs Lisp vectors are normally transmitted opaquely to Python. However, if pymacs-forget-mutability is set, or if Python later asks for these to be expanded, Emacs Lisp vectors get converted into Python tuples. In the other direction, Python tuples are always converted into Emacs Lisp vectors.

Remember the rule : Round parentheses correspond to square brackets!. It works for lists, vectors, tuples, seen from either Emacs Lisp or Python.

The above choices were debatable. Since Emacs Lisp proper lists and Python lists are the bread-and-butter of algorithms modifying structures, at least in my experience, I guess they are more naturally mapped into one another, this spares many casts in practice. While in Python, the most usual idiom for growing lists is appending to their end, the most usual idiom in Emacs Lisp to grow a list is by cons'ing new items at its beginning :

     (setq accumulator (cons 'new-item accumulator))


or more simply :

     (push 'new-item accumulator)


So, in case speed is especially important and many modifications happen in a row on the same side, while order of elements ought to be preserved, some (nreverse ...) on the Emacs Lisp side or .reverse() on the Python side side might be needed. Surely, proper lists in Emacs Lisp and lists in Python are the normal structure for which length is easily modified.

We cannot so easily change the size of a vector, the same as it is a bit more of a stunt to modify a tuple. The shape of these objects is fixed. Mapping vectors to tuples, which is admittedly strange, will only be done if the Python side requests an expanded copy, otherwise an opaque Emacs Lisp object is seen in Python. In the other direction, whenever an Emacs Lisp vector is needed, one has to write tuple(python_list) while transmitting the object. Such transmissions are most probably to be unusual, as people are not going to blindly transmit whole big structures back and forth between Emacs and Python, they would rather do it once in a while only, and do only local modifications afterwards. The infrequent casting to tuple for getting an Emacs Lisp vector seems to suggest that we did a reasonable compromise.

In Python, both tuples and lists have O(1) access, so there is no real speed consideration there. Emacs Lisp is different : vectors have O(1) access while lists have O(N) access. The rigidity of Emacs Lisp vectors is such that people do not resort to vectors unless there is a speed issue, so in real Emacs Lisp practice, vectors are used rather parsimoniously. So much, in fact, that Emacs Lisp vectors are overloaded for what they are not meant : for example, very small vectors are used to represent X events in key-maps, programmers only want to test vectors for their type, or users just like bracketed syntax. The speed of access is hardly an issue then.
.. @+node:ekr.20071103090504.5: *6* Opaque objects
.. @+node:ekr.20071103090504.6: *7* Emacs lisp handles
Emacs Lisp handles

When a Python function is called from Emacs Lisp, the function arguments have already been converted to Python types from Emacs Lisp types and the function result is going to be converted back to Emacs Lisp.

Several Emacs Lisp objects do not have Python equivalents, like for Emacs windows, buffers, markers, overlays, etc. It is nevertheless useful to pass them to Python functions, hoping that these Python functions will operate on these Emacs Lisp objects. Of course, the Python side may not itself modify such objects, it has to call for Emacs services to do so. Emacs Lisp handles are a mean to ease this communication.

Whenever an Emacs Lisp object may not be converted to a Python object, an Emacs Lisp handle is created and used instead. Whenever that Emacs Lisp handle is returned into Emacs Lisp from a Python function, or is used as an argument to an Emacs Lisp function from Python, the original Emacs Lisp object behind the Emacs Lisp handle is automatically retrieved.

Emacs Lisp handles are either instances of the internal Lisp class, or of one of its subclasses. If object is an Emacs Lisp handle, and if the underlying Emacs Lisp object is an Emacs Lisp sequence, then whenever object[index], object[index] = value and len(object) are meaningful, these may be used to fetch or alter an element of the sequence directly in Emacs Lisp space. Also, if object corresponds to an Emacs Lisp function, object(arguments) may be used to apply the Emacs Lisp function over the given arguments. Since arguments have been evaluated the Python way on the Python side, it would be conceptual overkill evaluating them again the Emacs Lisp way on the Emacs Lisp side, so Pymacs manage to quote arguments for defeating Emacs Lisp evaluation. The same logic applies the other way around.

Emacs Lisp handles have a value() method, which merely returns self. They also have a copy() method, which tries to open the box if possible. Emacs Lisp proper lists are turned into Python lists, Emacs Lisp vectors are turned into Python tuples. Then, modifying the structure of the copy on the Python side has no effect on the Emacs Lisp side.

For Emacs Lisp handles, str() returns an Emacs Lisp representation of the handle which should be eq to the original object if read back and evaluated in Emacs Lisp. repr() returns a Python representation of the expanded Emacs Lisp object. If that Emacs Lisp object has an Emacs Lisp representation which Emacs Lisp could read back, then repr() value is such that it could be read back and evaluated in Python as well, this would result in another object which is equal to the original, but not neccessarily eq.
.. @+node:ekr.20071103090504.7: *7* Python handles
Python handles

The same as Emacs Lisp handles are useful for handling Emacs Lisp objects on the Python side, Python handles are useful for handling Python objects on the Emacs Lisp side.

Many Python objects do not have direct Emacs Lisp equivalents, including long integers, complex numbers, Unicode strings, modules, classes, instances and surely a lot of others. When such are being transmitted to the Emacs Lisp side, Pymacs use Python handles. These are automatically recovered into the original Python objects whenever transmitted back to Python, either as arguments to a Python function, as the Python function itself, or as the return value of an Emacs Lisp function called from Python.

The objects represented by these Python handles may be inspected or modified using the basic library of Python functions. For example, in :

     (setq matcher (pymacs-eval "re.compile('pattern').match"))
     (pymacs-call matcher argument)


the initial setq above could be decomposed into :

           (setq compiled (pymacs-eval "re.compile('pattern')")
            matcher (pymacs-call "getattr" compiled "match"))


This example shows that one may use pymacs-call with getattr as the function, to get a wanted attribute for a Python object.
.. @+node:ekr.20071103090504.8: *6* Usages on the Emacs lisp side
.. @+node:ekr.20071103090504.9: *7* pymacs-eval/apply

pymacs-eval

Function (pymacs-eval text) gets text evaluated as a Python expression, and returns the value of that expression converted back to Emacs Lisp.

pymacs-call

Function (pymacs-call function argument...) will get Python to apply the given function over zero or more argument. function is either a string holding Python source code for a function (like a mere name, or even an expression), or else, a Python handle previously received from Python, and hopefully holding a callable Python object. Each argument gets separately converted to Python before the function is called. pymacs-call returns the resulting value of the function call, converted back to Emacs Lisp.

pymacs-apply

Function (pymacs-apply function arguments) will get Python to apply the given function over the given arguments. arguments is a list containing all arguments, or nil if there is none. Besides arguments being bundled together instead of given separately, the function acts pretty much like pymacs-call.

We do not expect that pymacs-eval, pymacs-call or pymacs-apply will be much used, if ever. In practice, the Emacs Lisp side of a Pymacs application might call pymacs-load a few times for linking into the Python modules, with the indirect effect of defining trampoline functions for these modules on the Emacs Lisp side, which can later be called like usual Emacs Lisp functions.
.. @+node:ekr.20071103090504.10: *7* pymacs-load
pymacs-load

Function (pymacs-load module prefix) imports the Python module into Emacs Lisp space.

module is the name of the file containing the module, without any .py or .pyc extension. If the directory part is omitted in module, the module will be looked into the current Python search path. Dot notation may be used when the module is part of a package. Each top-level function in the module produces a trampoline function in Emacs Lisp having the same name, except that underlines in Python names are turned into dashes in Emacs Lisp, and that prefix is uniformly added before the Emacs Lisp name (as a way to avoid name clashes).

prefix may be omitted, in which case it defaults to base name of module with underlines turned into dashes, and followed by a dash.

Whenever pymacs_load_hook is defined in the loaded Python module, pymacs-load calls it without arguments, but before creating the Emacs view for that module. So, the pymacs_load_hook function may create new definitions or even add interaction attributes to functions.

The return value of a successful pymacs-load is the module object. An optional third argument, noerror, when given and not nil, will have pymacs-load to return nil instead of raising an error, if the Python module could not be found.

When later calling one of these trampoline functions, all provided arguments are converted to Python and transmitted, and the function return value is later converted back to Emacs Lisp. It is left to the Python side to check for argument consistency. However, for an interactive function, the interaction specification drives some checking on the Emacs Lisp side. Currently, there is no provision for collecting keyword arguments in Emacs Lisp.
.. @+node:ekr.20071103091052: *6* Usage on the Python side
.. @+node:ekr.20071103091052.1: *7* Python setup
Python setup

Pymacs requires little or no setup in the Python modules which are meant to be used from Emacs, for the simple situations where these modules receive nothing but Emacs nil, numbers or strings, or return nothing but Python None, numbers or strings.

Otherwise, use from Pymacs import lisp. If you need more Pymacs features, like the Let class, write from Pymacs import lisp, Let.
.. @+node:ekr.20071103091052.2: *7* Response mode
Response mode

When Python receives a request from Emacs in the context of Pymacs, and until it returns the reply, Emacs keeps listening to serve Python requests. Emacs is not listening otherwise. Other Python threads, if any, may not call Emacs without very careful synchronisation.
.. @+node:ekr.20071103091052.3: *7* Emacs lisp symbols
Emacs Lisp symbols

lisp is a special object which has useful built-in magic. Its attributes do nothing but represent Emacs Lisp symbols, created on the fly as needed (symbols also have their built-in magic).

lisp.nil or lisp["nil"], are the same as None.

Otherwise, lisp.symbol and lisp[string] yield objects of the internal Symbol type. These are genuine Python objects, that could be referred to by simple Python variables. One may write quote = lisp.quote, for example, and use quote afterwards to mean that Emacs Lisp symbol. If a Python function received an Emacs Lisp symbol as an argument, it can check with == if that argument is lisp.never or lisp.ask, say. A Python function may well choose to return lisp.t.

In Python, writing lisp.symbol = value or lisp[string] = value does assign value to the corresponding symbol in Emacs Lisp space. Beware that in such cases, the lisp. prefix may not be [omitted] spared. After result = lisp.result, one cannot hope that a later result = 3 will have any effect in the Emacs Lisp space : this would merely change the Python variable result, which was a reference to a Symbol instance, so it is now a reference to the number 3.

The Symbol class has value() and copy() methods. One can use either lisp.symbol.value() or lisp.symbol.copy() to access the Emacs Lisp value of a symbol, after conversion to some Python object, of course. However, if value() would have given an Emacs Lisp handle, lisp.symbol.copy() has the effect of lisp.symbol.value().copy(), that is, it returns the value of the symbol as opened as possible.

A symbol may also be used as if it was a Python function, in which case it really names an Emacs Lisp function that should be applied over the following function arguments. The result of the Emacs Lisp function becomes the value of the call, with all due conversions of course.
.. @+node:ekr.20071103091052.4: *7* Dynamic bindings (The let class)
Dynamic bindings

As Emacs Lisp uses dynamic bindings, it is common that Emacs Lisp programs use
let for temporarily setting new values for some Emacs Lisp variables having
global scope. These variables recover their previous value automatically when
the let gets completed, even if an error occurs which interrupts the normal flow
of execution.

Pymacs has a Let class to represent such temporary settings. Suppose for example
that you want to recover the value of lisp.mark() when the transient mark mode
is active on the Emacs Lisp side. One could surely use lisp.mark(lisp.t) to
force reading the mark in such cases, but for the sake of illustration, let's
ignore that, and temporarily deactivate transient mark mode instead. This could
be done this way :

        try :
        let = Let()
        let.push(transient_mark_mode=None)
        ... user code ...
        finally :
        let.pop()

let.push() accepts any number of keywords arguments. Each keyword name is
interpreted as an Emacs Lisp symbol written the Pymacs way, with underlines. The
value of that Emacs Lisp symbol is saved on the Python side, and the value of
the keyword becomes the new temporary value for this Emacs Lisp symbol. A later
let.pop() restores the previous value for all symbols which were saved together
at the time of the corresponding let.push(). There may be more than one
let.push() call for a single Let instance, they stack within that instance. Each
let.pop() will undo one and only one let.push() from the stack, in the reverse
order or the pushes.

When the Let instance disappears, either because the programmer does del let or
let = None, or just because the Python let variable goes out of scope, all
remaining let.pop() get automatically executed, so the try/finally statement may
be omitted in practice. For this omission to work flawlessly, the programmer
should be careful at not keeping extra references to the Let instance.

The constructor call let = Let() also has an implied initial .push() over all
given arguments, so the explicit let.push() may be omitted as well. In practice,
this sums up and the above code could be reduced to a mere :

     let = Let(transient_mark_mode=None)
     ... user code ...

Be careful at assigning the result of the constructor to some Python variable.
Otherwise, the instance would disappear immediately after having been created,
restoring the Emacs Lisp variable much too soon.

Any variable to be bound with Let should have been bound in advance on the Emacs
Lisp side. This restriction usually does no kind of harm. Yet, it will likely be
lifted in some later version of Pymacs.

The Let class has other methods meant for some macros which are common in Emacs
Lisp programming, in the spirit of let bindings. These method names look like
push_* or pop_*, where Emacs Lisp macros are save-*. One has to use the matching
pop_* for undoing the effect of a given push_* rather than a mere .pop() : the
Python code is clearer, this also ensures that things are undone in the proper
order. The same Let instance may use many push_* methods, their effects nest.

push_excursion() and pop_excursion() save and restore the current buffer, point
and mark. push_match_data() and pop_match_data() save and restore the state of
the last regular expression match. push_restriction() and pop_restriction() save
and restore the current narrowing limits. push_selected_window() and
pop_selected_window() save and restore the fact that a window holds the cursor.
push_window_excursion() and pop_window_excursion() save and restore the current
window configuration in the Emacs display.

As a convenience, let.push() and all other push_* methods return the Let
instance. This helps chaining various push_* right after the instance
generation. For example, one may write :

         let = Let().push_excursion()
         if True :
         ... user code ...
         del let

The if True: (use if 1: with older Python releases, some people might prefer
writing if let: anyway), has the only goal of indenting user code, so the scope
of the let variable is made very explicit. This is purely stylistic, and not at
all necessary. The last del let might be omitted in a few circumstances, for
example if the excursion lasts until the end of the Python function.
.. @+node:ekr.20071103091052.5: *7* Raw Emacs lisp expression
Raw Emacs Lisp expressions

Pymacs offers a device for evaluating a raw Emacs Lisp expression, or a sequence of such, expressed as a string. One merely uses lisp as a function, like this :

     lisp("""
     ...
     possibly-long-sequence-of-lisp-expressions
     ...
     """)


The Emacs Lisp value of the last or only expression in the sequence becomes the value of the lisp call, after conversion back to Python.
.. @+node:ekr.20071103091052.6: *7* User interaction
User interaction

Emacs functions have the concept of user interaction for completing the specification of their arguments while being called. This happens only when a function is interactively called by the user, it does not happen when a function is programmatically called by another. As Python does not have a corresponding facility, a bit of trickery was needed to retrofit that facility on the Python side.

After loading a Python module but prior to creating an Emacs view for this
module, Pymacs decides whether loaded functions will be interactively callable
from Emacs, or not. Whenever a function has an interaction attribute, this
attribute holds the Emacs interaction specification for this function. The
specification is either another Python function or a string. In the former case,
that other function is called without arguments and should, maybe after having
consulted the user, return a list of the actual arguments to be used for the
original function. In the latter case, the specification string is used verbatim
as the argument to the (interactive ...) function on the Emacs side. To get a
short reminder about how this string is interpreted on the Emacs side, try C-h f
interactive within Emacs. Here is an example where an empty string is used to
specify that an interactive has no arguments::

    from Pymacs import lisp

    def hello_world() :
        "`Hello world' from Python."
        lisp.insert("Hello from Python!")
        hello_world.interaction = ''

Versions of Python released before the integration of PEP 232 do not allow users
to add attributes to functions, so there is a fallback mechanism. Let's presume
that a given function does not have an interaction attribute as explained above.
If the Python module contains an interactions global variable which is a
dictionary, if that dictionary has an entry for the given function with a value
other than None, that function is going to be interactive on the Emacs side.
Here is how the preceeding example should be written for an older version of
Python, or when portability is at premium::

    from Pymacs import lisp
    interactions = {}

    def hello_world() :
        "`Hello world' from Python."
        lisp.insert("Hello from Python!")
        interactions[hello_world] = ''

One might wonder why we do not merely use lisp.interactive(...) from within
Python. There is some magic in the Emacs Lisp interpreter itself, looking for
that call before the function is actually entered, this explains why
(interactive ...) has to appear first in an Emacs Lisp defun. Pymacs could try
to scan the already compiled form of the Python code, seeking for
lisp.interactive, but as the evaluation of lisp.interactive arguments could get
arbitrarily complex, it would a real challenge un-compiling that evaluation into
Emacs Lisp.
.. @+node:ekr.20071103091052.7: *7* Key bindings
Keybindings

An interactive function may be bound to a key sequence.

To translate bindings like C-x w, say, one might have to know a bit more how
Emacs Lisp processes string escapes like \C-x or \M-\C-x in Emacs Lisp, and
emulate it within Python strings, since Python does not have such escapes. \C-L,
where L is an upper case letter, produces a character which ordinal is the
result of subtracting 0x40 from ordinal of L. \M- has the ordinal one gets by
adding 0x80 to the ordinal of following described character. So people can use
self-inserting non-ASCII characters, \M- is given another representation, which
is to replace the addition of 0x80 by prefixing with `ESC', that is 0x1b.

So \C-x in Emacs is '\x18' in Python. This is easily found, using an interactive
Python session, by givin it : chr(ord('X') - ord('A') + 1). An easier way would
be using the kbd function on the Emacs Lisp side, like with lisp.kbd('C-x w') or
lisp.kbd('M-<f2>').

To bind the F1 key to the helper function in some module :

     lisp.global_set_key((lisp.f1,), lisp.module_helper)

(item,) is a Python tuple yielding an Emacs Lisp vector. lisp.f1 translates to
the Emacs Lisp symbol f1. So, Python (lisp.f1,) is Emacs Lisp [f1]. Keys like
[M-f2] might require some more ingenuity, one may write either (lisp['M-f2'],)
or (lisp.M_f2,) on the Python side.
.. @+node:ekr.20071103092153: *6* Debugging
.. @+node:ekr.20071103092153.1: *7* The *pymacs* buffer
The *Pymacs* buffer

Emacs and Python are two separate processes (well, each may use more than one process). Pymacs implements a simple communication protocol between both, and does whatever needed so the programmers do not have to worry about details. The main debugging tool is the communication buffer between Emacs and Python, which is named *Pymacs*. As it is sometimes helpful to understand the communication protocol, it is briefly explained here, using an artificially complex example to do so. Consider :

     (pymacs-eval "lisp('(pymacs-eval \"`2L**111`\")')")
     "2596148429267413814265248164610048L"

Here, Emacs asks Python to ask Emacs to ask Python for a simple bignum computation. Note that Emacs does not natively know how to handle big integers, nor has an internal representation for them. This is why I use backticks, so Python returns a string representation of the result, instead of the result itself. Here is a trace for this example. The < character flags a message going from Python to Emacs and is followed by an expression written in Emacs Lisp. The > character flags a message going from Emacs to Python and is followed by a expression written in Python. The number gives the length of the message.

     <22   (pymacs-version "0.3")
     >49   eval("lisp('(pymacs-eval \"`2L**111`\")')")
     <25   (pymacs-eval "`2L**111`")
     >18   eval("`2L**111`")
     <47   (pymacs-reply "2596148429267413814265248164610048L")
     >45   reply("2596148429267413814265248164610048L")
     <47   (pymacs-reply "2596148429267413814265248164610048L")

Python evaluation is done in the context of the Pymacs.pymacs module, so for example a mere reply really means Pymacs.pymacs.reply. On the Emacs Lisp side, there is no concept of module namespaces, so we use the pymacs- prefix as an attempt to stay clean. Users should ideally refrain from naming their Emacs Lisp objects with a pymacs- prefix.

reply and pymacs-reply are special functions meant to indicate that an expected result is finally transmitted. error and pymacs-error are special functions that introduce a string which explains an exception which recently occurred. pymacs-expand is a special function implementing the copy() methods of Emacs Lisp handles or symbols. In all other cases, the expression is a request for the other side, that request stacks until a corresponding reply is received.

Part of the protocol manages memory, and this management generates some extra-noise in the *Pymacs* buffer. Whenever Emacs passes a structure to Python, an extra pointer is generated on the Emacs side to inhibit garbage collection by Emacs. Python garbage collector detects when the received structure is no longer needed on the Python side, at which time the next communication will tell Emacs to remove the extra pointer. It works symmetrically as well, that is, whenever Python passes a structure to Emacs, an extra Python reference is generated to inhibit garbage collection on the Python side. Emacs garbage collector detects when the received structure is no longer needed on the Emacs side, after which Python will be told to remove the extra reference. For efficiency, those allocation-related messages are delayed, merged and batched together within the next communication having another purpose.

Variable pymacs-trace-transit may be modified for controlling how and when the *Pymacs* buffer, or parts thereof, get erased.
.. @+node:ekr.20071103092153.2: *7* Usual Emacs debugging
Emacs usual debugging

If cross-calls between Emacs Lisp and Python nest deeply, an error will raise
successive exceptions alternatively on both sides as requests unstack, and the
diagnostic gets transmitted back and forth, slightly growing as we go. So,
errors will eventually be reported by Emacs. I made no kind of effort to
transmit the Emacs Lisp backtrace on the Python side, as I do not see a purpose
for it : all debugging is done within Emacs windows anyway.

On recent Emacses, the Python backtrace gets displayed in the mini-buffer, and
the Emacs Lisp backtrace is simultaneously shown in the *Backtrace* window. One
useful thing is to allow to mini-buffer to grow big, so it has more chance to
fully contain the Python backtrace, the last lines of which are often especially
useful. Here, I use :

         (setq resize-mini-windows t
          max-mini-window-height .85)

in my .emacs file, so the mini-buffer may use 85% of the screen, and quickly
shrinks when fewer lines are needed. The mini-buffer contents disappear at the
next keystroke, but you can recover the Python backtrace by looking at the end
of the *Messages* buffer. In which case the ffap package in Emacs may be yet
another friend! From the *Messages* buffer, once ffap activated, merely put the
cursor on the file name of a Python module from the backtrace, and C-x C-f RET
will quickly open that source for you.
.. @+node:ekr.20071103092153.3: *7* Auto-reloading on save
Auto-reloading on save

I found useful to automatically pymacs-load some Python files whenever they get
saved from Emacs. This can be decided on a per-file or per-directory basis. To
get a particular Python file to be reloaded automatically on save, add the
following lines at the end :

     # Local Variables :
     # pymacs-auto-reload : t
     # End :

Here is an example of automatic reloading on a per-directory basis. The code
below assumes that Python files meant for Pymacs are kept in
~/share/emacs/python.

    (defun fp-maybe-pymacs-reload ()
        (let ((pymacsdir (expand-file-name "~/share/emacs/python/")))
         (when (and (string-equal (file-name-directory buffer-file-name)
                  pymacsdir)
              (string-match "\\.py\\'" buffer-file-name))
          (pymacs-load (substring buffer-file-name 0 -3)))))
         (add-hook 'after-save-hook 'fp-maybe-pymacs-reload)
.. @+node:ekr.20071103092153.4: *6* Example 1: defining an Emacs command in Python
@language rest

Let's say I have a a module, call it manglers.py, containing this simple python
function::

    def break_on_whitespace(some_string) :
         words = some_string.split()
         return '\n'.join(words)

The goal is telling Emacs about this function so that I can call it on a region
of text and replace the region with the result of the call. We shall also bind
this function to the key [f7].

Here is the Python side::

    from Pymacs import lisp
    interactions = {}

    def break_on_whitespace():
        # start and end may be given in any order.
        start,end = lisp.point(),lisp.mark(lisp.t)
        words = lisp.buffer_substring(start, end).split()
        lisp.delete_region(start,end)
        lisp.insert('\n'.join(words))

    interactions[break_on_whitespace] = ''

Here is the emacs side::

    (pymacs-load "manglers")
    (global-set-key [f7] 'manglers-break-on-whitespace)
.. @+node:ekr.20071103093725: *6* Example 3: defining a rebox tool
For comments held within boxes, it is painful to fill paragraphs, while
stretching or shrinking the surrounding box by hand, as needed. This piece of
Python code eases my life on this. It may be used interactively from within
Emacs through the Pymacs interface, or in batch as a script which filters a
single region to be reformatted.

In batch mode, the reboxing is driven by command options and arguments and expects a
complete, self-contained boxed comment from a file.

Emacs function rebox-region also presumes that the region encloses a single
boxed comment.

Emacs rebox-comment is different, as it has to chase itself the extent of the
surrounding boxed comment.

.. @+node:ekr.20071103094355: *7* The python side
Design notes for rebox.py:

Pymacs specific features are used exclusively from within the pymacs_load_hook
function and the Emacs_Rebox class. In batch mode, Pymacs is not even imported.

In batch mode, as well as with rebox-region, the text to handle is turned over
to Python, and fully processed in Python, with practically no Pymacs interaction
while the work gets done. On the other hand, rebox-comment is rather Pymacs
intensive: the comment boundaries are chased right from the Emacs buffer, as
directed by the function Emacs_Rebox.find_comment. Once the boundaries are
found, the remainder of the work is essentially done on the Python side.

Once the boxed comment has been reformatted in Python, the old comment is
removed in a single delete operation, the new comment is inserted in a second
operation. This occurs in Emacs_Rebox.process_emacs_region. But by doing so, if
point was within the boxed comment before the reformatting, its precise position
is lost. To well preserve point, Python might have driven all reformatting
details directly in the Emacs buffer. We really preferred doing it all on the
Python side : as we gain legibility by expressing the algorithms in pure Python,
the same Python code may be used in batch or interactively, and we avoid the
slowdown that would result from heavy use of Emacs services.

To avoid completely loosing point, I kludged a Marker class, which goal is to
estimate the new value of point from the old. Reformatting may change the amount
of white space, and either delete or insert an arbitrary number characters meant
to draw the box. The idea is to initially count the number of characters between
the beginning of the region and point, while ignoring any problematic character.
Once the comment has been reboxed, point is advanced from the beginning of the
region until we get the same count of characters, skipping all problematic
characters. This Marker class works fully on the Python side, it does not
involve Pymacs at all, but it does solve a problem that resulted from my choice
of keeping the data on the Python side instead of handling it directly in the
Emacs buffer.

We want a comment reformatting to appear as a single operation, in the context
of Emacs Undo. The method Emacs_Rebox.clean_undo_after handles the general case
for this. Not that we do so much in practice : a reformatting implies one
delete-region and one insert, and maybe some other little adjustements at
Emacs_Rebox.find_comment time. Even if this method scans and mofifies an Emacs
Lisp list directly in the Emacs memory, the code doing this stays neat and
legible. However, I found out that the undo list may grow quickly when the Emacs
buffer use markers, with the consequence of making this routine so Pymacs
intensive that most of the CPU is spent there. I rewrote that routine in Emacs
Lisp so it executes in a single Pymacs interaction.

Function Emacs_Rebox.remainder_of_line could have been written in Python, but it
was probably not worth going away from this one-liner in Emacs Lisp. Also, given
this routine is often called by find_comment, a few Pymacs protocol interactions
are spared this way. This function is useful when there is a need to apply a
regexp already compiled on the Python side, it is probably better fetching the
line from Emacs and do the pattern match on the Python side, than transmitting
the source of the regexp to Emacs for it to compile and apply it.

For refilling, I could have either used the refill algorithm built within in
Emacs, programmed a new one in Python, or relied on Ross Paterson's fmt,
distributed by GNU and available on most Linuxes. In fact, refill_lines prefers
the latter. My own Emacs setup is such that the built-in refill algorithm is
already overridden by GNU fmt, and it really does a much better job. Experience
taught me that calling an external program is fast enough to be very bearable,
even interactively. If Python called Emacs to do the refilling, Emacs would
itself call GNU fmt in my case, I preferred that Python calls GNU fmt directly.
I could have reprogrammed GNU fmt in Python. Despite interesting, this is an
uneasy project : fmt implements the Knuth refilling algorithm, which depends on
dynamic programming techniques; Ross did carefully fine tune them, and took care
of many details. If GNU fmt fails, for not being available, say, refill_lines
falls back on a dumb refilling algorithm, which is better than none.
.. @+node:ekr.20071103094355.1: *7* The emacs side
For most Emacs language editing modes, refilling does not make sense
outside comments, one may redefine the `M-q' command and link it to this
Pymacs module.  For example, I use this in my `.emacs' file::

    (add-hook 'c-mode-hook 'fp-c-mode-routine)
    (defun fp-c-mode-routine ()
        (local-set-key "\M-q" 'rebox-comment))
    (autoload 'rebox-comment "rebox" nil t)
    (autoload 'rebox-region "rebox" nil t)

with a "rebox.el" file having this single line:

    (pymacs-load "Pymacs.rebox")

The Emacs function `rebox-comment' automatically discovers the extent of the
boxed comment near the cursor, possibly refills the text, then adjusts the box
style. When this command is executed, the cursor should be within a comment, or
else it should be between two comments, in which case the command applies to the
next comment.

The Emacs function `rebox-region' does the same, except that it takes the
current region as a boxed comment. Both commands obey numeric prefixes to add or
remove a box, force a particular box style, or to prevent refilling of text.
Without such prefixes, the commands may deduce the current box style from the
comment itself so the style is preserved.

The default style initial value is nil or 0.  It may be preset to another
value through calling `rebox-set-default-style' from Emacs LISP, or changed
to anything else though using a negative value for a prefix, in which case
the default style is set to the absolute value of the prefix.

A `C-u' prefix avoids refilling the text, but forces using the default box
style.  `C-u -' lets the user interact to select one attribute at a time.

Adding new styles
-----------------

Let's suppose you want to add your own boxed comment style, say:

    //--------------------------------------------+
    // This is the style mandated in our company.
    //--------------------------------------------+

You might modify `rebox.py' but then, you will have to edit it whenever you
get a new release of `pybox.py'.  Emacs users might modify their `.emacs'
file or their `rebox.el' bootstrap, if they use one.  In either cases,
after the `(pymacs-load "Pymacs.rebox")' line, merely add:

    (rebox-Template NNN MMM ["//-----+"
                             "// box  "
                             "//-----+"])

In batch mode [If you use the `rebox' script rather than Emacs], the simplest is to make
your own.  This is easy, as it is very small.  For example, the above
style could be implemented by using this script instead of `rebox':

    #!/usr/bin/env python
    import sys
    from Pymacs import rebox
    rebox.Template(226, 325, ('//-----+',
                              '// box  ',
                              '//-----+'))
    apply(rebox.main, tuple(sys.argv[1:]))

In all cases, NNN is the style three-digit number, with no zero digit.
Pick any free style number, you are safe with 911 and up.  MMM is the
recognition priority, only used to disambiguate the style of a given boxed
comments, when it matches many styles at once.  Try something like 400.
Raise or lower that number as needed if you observe false matches.

Usually, the template uses three lines of equal length.  Do not worry if
this implies a few trailing spaces, they will be cleaned up automatically
at box generation time.  The first line or the third line may be omitted
to create vertically opened boxes.  But the middle line may not be omitted,
it ought to include the word `box', which will get replaced by your actual
comment.  If the first line is shorter than the middle one, it gets merged
at the start of the comment.  If the last line is shorter than the middle
one, it gets merged at the end of the comment and is refilled with it.
.. @+node:ekr.20071103093725.1: *7* rebox.py
@color
@language python
@tabwidth -4

@others
<< templates >>

if __name__ == '__main__':
    apply(main, sys.argv[1:])
.. @+node:ekr.20071103093725.2: *8* rebox declarations
#!/usr/bin/env python
# Copyright © 1991-1998, 2000, 2002 Progiciels Bourbeau-Pinard inc.
# François Pinard <pinard@iro.umontreal.ca>, April 1991.

# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.

# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software Foundation,
# Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.

"""\
Handling of boxed comments in various box styles.

Introduction
------------

For comments held within boxes, it is painful to fill paragraphs, while
stretching or shrinking the surrounding box "by hand", as needed.  This piece
of Python code eases my life on this.  It may be used interactively from
within Emacs through the Pymacs interface, or in batch as a script which
filters a single region to be reformatted.  I find only fair, while giving
all sources for a package using such boxed comments, to also give the
means I use for nicely modifying comments.  So here they are!

Box styles
----------

Each supported box style has a number associated with it.  This number is
arbitrary, yet by _convention_, it holds three non-zero digits such the the
hundreds digit roughly represents the programming language, the tens digit
roughly represents a box quality (or weight) and the units digit roughly
a box type (or figure).  An unboxed comment is merely one of box styles.
Language, quality and types are collectively referred to as style attributes.

When rebuilding a boxed comment, attributes are selected independently
of each other.  They may be specified by the digits of the value given
as Emacs commands argument prefix, or as the `-s' argument to the `rebox'
script when called from the shell.  If there is no such prefix, or if the
corresponding digit is zero, the attribute is taken from the value of the
default style instead.  If the corresponding digit of the default style
is also zero, than the attribute is recognised and taken from the actual
boxed comment, as it existed before prior to the command.  The value 1,
which is the simplest attribute, is ultimately taken if the parsing fails.

A programming language is associated with comment delimiters.  Values are
100 for none or unknown, 200 for `/*' and `*/' as in plain C, 300 for `//'
as in C++, 400 for `#' as in most scripting languages, 500 for `;' as in
LISP or assembler and 600 for `%' as in TeX or PostScript.

Box quality differs according to language. For unknown languages (100) or
for the C language (200), values are 10 for simple, 20 for rounded, and
30 or 40 for starred.  Simple quality boxes (10) use comment delimiters
to left and right of each comment line, and also for the top or bottom
line when applicable. Rounded quality boxes (20) try to suggest rounded
corners in boxes.  Starred quality boxes (40) mostly use a left margin of
asterisks or X'es, and use them also in box surroundings.  For all others
languages, box quality indicates the thickness in characters of the left
and right sides of the box: values are 10, 20, 30 or 40 for 1, 2, 3 or 4
characters wide.  With C++, quality 10 is not useful, it is not allowed.

Box type values are 1 for fully opened boxes for which boxing is done
only for the left and right but not for top or bottom, 2 for half
single lined boxes for which boxing is done on all sides except top,
3 for fully single lined boxes for which boxing is done on all sides,
4 for half double lined boxes which is like type 2 but more bold,
or 5 for fully double lined boxes which is like type 3 but more bold.

The special style 221 is for C comments between a single opening `/*'
and a single closing `*/'.  The special style 111 deletes a box.

Batch usage
-----------

Usage is `rebox [OPTION]... [FILE]'.  By default, FILE is reformatted to
standard output by refilling the comment up to column 79, while preserving
existing boxed comment style.  If FILE is not given, standard input is read.
Options may be:

  -n         Do not refill the comment inside its box, and ignore -w.
  -s STYLE   Replace box style according to STYLE, as explained above.
  -t         Replace initial sequence of spaces by TABs on each line.
  -v         Echo both the old and the new box styles on standard error.
  -w WIDTH   Try to avoid going over WIDTH columns per line.

So, a single boxed comment is reformatted by invocation.  `vi' users, for
example, would need to delimit the boxed comment first, before executing
the `!}rebox' command (is this correct? my `vi' recollection is far away).

Batch usage is also slow, as internal structures have to be reinitialised
at every call.  Producing a box in a single style is fast, but recognising
the previous style requires setting up for all possible styles.

Emacs usage
-----------

For most Emacs language editing modes, refilling does not make sense
outside comments, one may redefine the `M-q' command and link it to this
Pymacs module.  For example, I use this in my `.emacs' file:

     (add-hook 'c-mode-hook 'fp-c-mode-routine)
     (defun fp-c-mode-routine ()
       (local-set-key "\M-q" 'rebox-comment))
     (autoload 'rebox-comment "rebox" nil t)
     (autoload 'rebox-region "rebox" nil t)

with a "rebox.el" file having this single line:

     (pymacs-load "Pymacs.rebox")

Install Pymacs from `http://www.iro.umontreal.ca/~pinard/pymacs.tar.gz'.

The Emacs function `rebox-comment' automatically discovers the extent of
the boxed comment near the cursor, possibly refills the text, then adjusts
the box style.  When this command is executed, the cursor should be within
a comment, or else it should be between two comments, in which case the
command applies to the next comment.  The function `rebox-region' does
the same, except that it takes the current region as a boxed comment.
Both commands obey numeric prefixes to add or remove a box, force a
particular box style, or to prevent refilling of text.  Without such
prefixes, the commands may deduce the current box style from the comment
itself so the style is preserved.

The default style initial value is nil or 0.  It may be preset to another
value through calling `rebox-set-default-style' from Emacs LISP, or changed
to anything else though using a negative value for a prefix, in which case
the default style is set to the absolute value of the prefix.

A `C-u' prefix avoids refilling the text, but forces using the default box
style.  `C-u -' lets the user interact to select one attribute at a time.

Adding new styles
-----------------

Let's suppose you want to add your own boxed comment style, say:

    //--------------------------------------------+
    // This is the style mandated in our company.
    //--------------------------------------------+

You might modify `rebox.py' but then, you will have to edit it whenever you
get a new release of `pybox.py'.  Emacs users might modify their `.emacs'
file or their `rebox.el' bootstrap, if they use one.  In either cases,
after the `(pymacs-load "Pymacs.rebox")' line, merely add:

    (rebox-Template NNN MMM ["//-----+"
                             "// box  "
                             "//-----+"])

If you use the `rebox' script rather than Emacs, the simplest is to make
your own.  This is easy, as it is very small.  For example, the above
style could be implemented by using this script instead of `rebox':

    #!/usr/bin/env python
    import sys
    from Pymacs import rebox
    rebox.Template(226, 325, ('//-----+',
                              '// box  ',
                              '//-----+'))
    apply(rebox.main, tuple(sys.argv[1:]))

In all cases, NNN is the style three-digit number, with no zero digit.
Pick any free style number, you are safe with 911 and up.  MMM is the
recognition priority, only used to disambiguate the style of a given boxed
comments, when it matches many styles at once.  Try something like 400.
Raise or lower that number as needed if you observe false matches.

On average, the template uses three lines of equal length.  Do not worry if
this implies a few trailing spaces, they will be cleaned up automatically
at box generation time.  The first line or the third line may be omitted
to create vertically opened boxes.  But the middle line may not be omitted,
it ought to include the word `box', which will get replaced by your actual
comment.  If the first line is shorter than the middle one, it gets merged
at the start of the comment.  If the last line is shorter than the middle
one, it gets merged at the end of the comment and is refilled with it.

History
-------

I first observed rounded corners, as in style 223 boxes, in code from
Warren Tucker, a previous maintainer of the `shar' package, circa 1980.

Except for very special files, I carefully avoided boxed comments for
real work, as I found them much too hard to maintain.  My friend Paul
Provost was working at Taarna, a computer graphics place, which had boxes
as part of their coding standards.  He asked that we try something to get
him out of his misery, and this how `rebox.el' was originally written.
I did not plan to use it for myself, but Paul was so enthusiastic that I
timidly started to use boxes in my things, very little at first, but more
and more as time passed, still in doubt that it was a good move.  Later,
many friends spontaneously started to use this tool for real, some being very
serious workers.  This convinced me that boxes are acceptable, after all.

I do not use boxes much with Python code.  It is so legible that boxing
is not that useful.  Vertical white space is less necessary, too.  I even
avoid white lines within functions.  Comments appear prominent enough when
using highlighting editors like Emacs or nice printer tools like `enscript'.

After Emacs could be extended with Python, in 2001, I translated `rebox.el'
into `rebox.py', and added the facility to use it as a batch script.
"""

## Note: This code is currently compatible down to Python version 1.5.2.
## It is probably worth keeping it that way for a good while, still.

## Note: a double hash comment introduces a group of functions or methods.

import re, string, sys

.. @+node:ekr.20071103093725.3: *8* main
def main(*arguments):
    refill = 1
    style = None
    tabify = 0
    verbose = 0
    width = 79
    import getopt
    options, arguments = getopt.getopt(arguments, 'ns:tvw:', ['help'])
    for option, value in options:
        if option == '--help':
            sys.stdout.write(__doc__)
            sys.exit(0)
        elif option == '-n':
            refill = 0
        elif option == '-s':
            style = int(value)
        elif option == '-t':
            tabify = 1
        elif option == '-v':
            verbose = 1
        elif option == '-w':
            width = int(value)
    if len(arguments) == 0:
        text = sys.stdin.read()
    elif len(arguments) == 1:
        text = open(arguments[0]).read()
    else:
        sys.stderr.write("Invalid usage, try `rebox --help' for help.\n")
        sys.exit(1)
    old_style, new_style, text, position = engine(
        text, style=style, width=width, refill=refill, tabify=tabify)
    if text is None:
        sys.stderr.write("* Cannot rebox to style %d.\n" % new_style)
        sys.exit(1)
    sys.stdout.write(text)
    if verbose:
        if old_style == new_style:
            sys.stderr.write("Reboxed with style %d.\n" % old_style)
        else:
            sys.stderr.write("Reboxed from style %d to %d.\n"
                             % (old_style, new_style))

.. @+node:ekr.20071103093725.4: *8* pymacs_load_hook
def pymacs_load_hook():
    global interactions, lisp, Let, region, comment, set_default_style
    from Pymacs import lisp, Let
    emacs_rebox = Emacs_Rebox()
    # Declare functions for Emacs to import.
    interactions = {}
    region = emacs_rebox.region
    interactions[region] = 'P'
    comment = emacs_rebox.comment
    interactions[comment] = 'P'
    set_default_style = emacs_rebox.set_default_style

.. @+node:ekr.20071103093725.5: *8* class Emacs_Rebox
class Emacs_Rebox:
    @others
.. @+node:ekr.20071103093725.6: *9* __init__ (Emacs_Rebox)

def __init__(self):
    self.default_style = None

.. @+node:ekr.20071103093725.7: *9* set_default_style
def set_default_style(self, style):
    """\
Set the default style to STYLE.
"""
    self.default_style = style

.. @+node:ekr.20071103093725.8: *9* region
def region(self, flag):
    """\
Rebox the boxed comment in the current region, obeying FLAG.
"""
    self.emacs_engine(flag, self.find_region)

.. @+node:ekr.20071103093725.9: *9* comment
def comment(self, flag):
    """\
Rebox the surrounding boxed comment, obeying FLAG.
"""
    self.emacs_engine(flag, self.find_comment)

.. @+node:ekr.20071103093725.10: *9* emacs_engine
def emacs_engine(self, flag, find_limits):
    """\
Rebox text while obeying FLAG.  Call FIND_LIMITS to discover the extent
of the boxed comment.
"""
    # `C-u -' means that box style is to be decided interactively.
    if flag == lisp['-']:
        flag = self.ask_for_style()
    # If FLAG is zero or negative, only change default box style.
    if type(flag) is type(0) and flag <= 0:
        self.default_style = -flag
        lisp.message("Default style set to %d" % -flag)
        return
    # Decide box style and refilling.
    if flag is None:
        style = self.default_style
        refill = 1
    elif type(flag) == type(0):
        if self.default_style is None:
            style = flag
        else:
            style = merge_styles(self.default_style, flag)
        refill = 1
    else:
        flag = flag.copy()
        if type(flag) == type([]):
            style = self.default_style
            refill = 0
        else:
            lisp.error("Unexpected flag value %s" % flag)
    # Prepare for reboxing.
    lisp.message("Reboxing...")
    checkpoint = lisp.buffer_undo_list.value()
    start, end = find_limits()
    text = lisp.buffer_substring(start, end)
    width = lisp.fill_column.value()
    tabify = lisp.indent_tabs_mode.value() is not None
    point = lisp.point()
    if start <= point < end:
        position = point - start
    else:
        position = None
    # Rebox the text and replace it in Emacs buffer.
    old_style, new_style, text, position = engine(
        text, style=style, width=width,
        refill=refill, tabify=tabify, position=position)
    if text is None:
        lisp.error("Cannot rebox to style %d" % new_style)
    lisp.delete_region(start, end)
    lisp.insert(text)
    if position is not None:
        lisp.goto_char(start + position)
    # Collapse all operations into a single one, for Undo.
    self.clean_undo_after(checkpoint)
    # We are finished, tell the user.
    if old_style == new_style:
        lisp.message("Reboxed with style %d" % old_style)
    else:
        lisp.message("Reboxed from style %d to %d"
                     % (old_style, new_style))

.. @+node:ekr.20071103093725.11: *9* ask_for_style
def ask_for_style(self):
    """\
Request the style interactively, using the minibuffer.
"""
    language = quality = type = None
    while language is None:
        lisp.message("\
Box language is 100-none, 200-/*, 300-//, 400-#, 500-;, 600-%%")
        key = lisp.read_char()
        if key >= ord('0') and key <= ord('6'):
            language = key - ord('0')
    while quality is None:
        lisp.message("\
Box quality/width is 10-simple/1, 20-rounded/2, 30-starred/3 or 40-starred/4")
        key = lisp.read_char()
        if key >= ord('0') and key <= ord('4'):
            quality = key - ord('0')
    while type is None:
        lisp.message("\
Box type is 1-opened, 2-half-single, 3-single, 4-half-double or 5-double")
        key = lisp.read_char()
        if key >= ord('0') and key <= ord('5'):
            type = key - ord('0')
    return 100*language + 10*quality + type

.. @+node:ekr.20071103093725.12: *9* find_region
def find_region(self):
    """\
Return the limits of the region.
"""
    return lisp.point(), lisp.mark(lisp.t)

.. @+node:ekr.20071103093725.13: *9* find_comment
def find_comment(self):
    """\
Find and return the limits of the block of comments following or enclosing
the cursor, or return an error if the cursor is not within such a block
of comments.  Extend it as far as possible in both directions.
"""
    let = Let()
    let.push_excursion()
    # Find the start of the current or immediately following comment.
    lisp.beginning_of_line()
    lisp.skip_chars_forward(' \t\n')
    lisp.beginning_of_line()
    if not language_matcher[0](self.remainder_of_line()):
        temp = lisp.point()
        if not lisp.re_search_forward('\\*/', None, lisp.t):
            lisp.error("outside any comment block")
        lisp.re_search_backward('/\\*')
        if lisp.point() > temp:
            lisp.error("outside any comment block")
        temp = lisp.point()
        lisp.beginning_of_line()
        lisp.skip_chars_forward(' \t')
        if lisp.point() != temp:
            lisp.error("text before start of comment")
        lisp.beginning_of_line()
    start = lisp.point()
    language = guess_language(self.remainder_of_line())
    # Find the end of this comment.
    if language == 2:
        lisp.search_forward('*/')
        if not lisp.looking_at('[ \t]*$'):
            lisp.error("text after end of comment")
    lisp.end_of_line()
    if lisp.eobp():
        lisp.insert('\n')
    else:
        lisp.forward_char(1)
    end = lisp.point()
    # Try to extend the comment block backwards.
    lisp.goto_char(start)
    while not lisp.bobp():
        if language == 2:
            lisp.skip_chars_backward(' \t\n')
            if not lisp.looking_at('[ \t]*\n[ \t]*/\\*'):
                break
            if lisp.point() < 2:
                break
            lisp.backward_char(2)
            if not lisp.looking_at('\\*/'):
                break
            lisp.re_search_backward('/\\*')
            temp = lisp.point()
            lisp.beginning_of_line()
            lisp.skip_chars_forward(' \t')
            if lisp.point() != temp:
                break
            lisp.beginning_of_line()
        else:
            lisp.previous_line(1)
            if not language_matcher[language](self.remainder_of_line()):
                break
        start = lisp.point()
    # Try to extend the comment block forward.
    lisp.goto_char(end)
    while language_matcher[language](self.remainder_of_line()):
        if language == 2:
            lisp.re_search_forward('[ \t]*/\\*')
            lisp.re_search_forward('\\*/')
            if lisp.looking_at('[ \t]*$'):
                lisp.beginning_of_line()
                lisp.forward_line(1)
                end = lisp.point()
        else:
            lisp.forward_line(1)
            end = lisp.point()
    return start, end

.. @+node:ekr.20071103093725.14: *9* remainder_of_line
def remainder_of_line(self):
    """\
Return all characters between point and end of line in Emacs buffer.
"""
    return lisp('''\
(buffer-substring (point) (save-excursion (skip-chars-forward "^\n") (point)))
''')

.. @+node:ekr.20071103093725.15: *9* clean_undo_after_old
def clean_undo_after_old(self, checkpoint):
    """\
Remove all intermediate boundaries from the Undo list since CHECKPOINT.
"""
    # Declare some LISP functions.
    car = lisp.car
    cdr = lisp.cdr
    eq = lisp.eq
    setcdr = lisp.setcdr
    # Remove any `nil' delimiter recently added to the Undo list.
    cursor = lisp.buffer_undo_list.value()
    if not eq(cursor, checkpoint):
        tail = cdr(cursor)
        while not eq(tail, checkpoint):
            if car(tail):
                cursor = tail
                tail = cdr(cursor)
            else:
                tail = cdr(tail)
                setcdr(cursor, tail)

.. @+node:ekr.20071103093725.16: *9* clean_undo_after
def clean_undo_after(self, checkpoint):
    """\
Remove all intermediate boundaries from the Undo list since CHECKPOINT.
"""
    lisp("""
(let ((undo-list %s))
(if (not (eq buffer-undo-list undo-list))
  (let ((cursor buffer-undo-list))
(while (not (eq (cdr cursor) undo-list))
  (if (car (cdr cursor))
      (setq cursor (cdr cursor))
    (setcdr cursor (cdr (cdr cursor)))))))
nil)
"""
         % (checkpoint or 'nil'))

.. @+node:ekr.20071103093725.17: *8* engine
def engine(text, style=None, width=79, refill=1, tabify=0, position=None):
    """\
Add, delete or adjust a boxed comment held in TEXT, according to STYLE.
STYLE values are explained at beginning of this file.  Any zero attribute
in STYLE indicates that the corresponding attribute should be recovered
from the currently existing box.  Produced lines will not go over WIDTH
columns if possible, if refilling gets done.  But if REFILL is false, WIDTH
is ignored.  If TABIFY is true, the beginning of produced lines will have
spaces replace by TABs.  POSITION is either None, or a character position
within TEXT.  Returns four values: the old box style, the new box style,
the reformatted text, and either None or the adjusted value of POSITION in
the new text.  The reformatted text is returned as None if the requested
style does not exist.
"""
    last_line_complete = text and text[-1] == '\n'
    if last_line_complete:
        text = text[:-1]
    lines = string.split(string.expandtabs(text), '\n')
    # Decide about refilling and the box style to use.
    new_style = 111
    old_template = guess_template(lines)
    new_style = merge_styles(new_style, old_template.style)
    if style is not None:
        new_style = merge_styles(new_style, style)
    new_template = template_registry.get(new_style)
    # Interrupt processing if STYLE does not exist.
    if not new_template:
        return old_template.style, new_style, None, None
    # Remove all previous comment marks, and left margin.
    if position is not None:
        marker = Marker()
        marker.save_position(text, position, old_template.characters())
    lines, margin = old_template.unbuild(lines)
    # Ensure only one white line between paragraphs.
    counter = 1
    while counter < len(lines) - 1:
        if lines[counter] == '' and lines[counter-1] == '':
            del lines[counter]
        else:
            counter = counter + 1
    # Rebuild the boxed comment.
    lines = new_template.build(lines, width, refill, margin)
    # Retabify to the left only.
    if tabify:
        for counter in range(len(lines)):
            tabs = len(re.match(' *', lines[counter]).group()) / 8
            lines[counter] = '\t' * tabs + lines[counter][8*tabs:]
    # Restore the point position.
    text = string.join(lines, '\n')
    if last_line_complete:
        text = text + '\n'
    if position is not None:
        position = marker.get_position(text, new_template.characters())
    return old_template.style, new_style, text, position

.. @+node:ekr.20071103093725.18: *8* guess_language
def guess_language(line):
    """\
Guess the language in use for LINE.
"""
    for language in range(len(language_matcher) - 1, 1, -1):
        if language_matcher[language](line):
            return language
    return 1

.. @+node:ekr.20071103093725.19: *8* guess_template
def guess_template(lines):
    """\
Find the heaviest box template matching LINES.
"""
    best_template = None
    for template in template_registry.values():
        if best_template is None or template > best_template:
            if template.match(lines):
                best_template = template
    return best_template

.. @+node:ekr.20071103093725.20: *8* left_margin_size
def left_margin_size(lines):
    """\
Return the width of the left margin for all LINES.  Ignore white lines.
"""
    margin = None
    for line in lines:
        counter = len(re.match(' *', line).group())
        if counter != len(line):
            if margin is None or counter < margin:
                margin = counter
    if margin is None:
        margin = 0
    return margin

.. @+node:ekr.20071103093725.21: *8* merge_styles
def merge_styles(original, update):
    """\
Return style attributes as per ORIGINAL, in which attributes have been
overridden by non-zero corresponding style attributes from UPDATE.
"""
    style = [original / 100, original / 10 % 10, original % 10]
    merge = update / 100, update / 10 % 10, update % 10
    for counter in range(3):
        if merge[counter]:
            style[counter] = merge[counter]
    return 100*style[0] + 10*style[1] + style[2]

.. @+node:ekr.20071103093725.22: *8* refill_lines
def refill_lines(lines, width):
    """\
Refill LINES, trying to not produce lines having more than WIDTH columns.
"""
    # Try using GNU `fmt'.
    import tempfile, os
    name = tempfile.mktemp()
    open(name, 'w').write(string.join(lines, '\n') + '\n')
    process = os.popen('fmt -cuw %d %s' % (width, name))
    text = process.read()
    os.remove(name)
    if process.close() is None:
        return map(string.expandtabs, string.split(text, '\n')[:-1])
    # If `fmt' failed, do refilling more naively, wihtout using the
    # Knuth algorithm, nor protecting full stops at end of sentences.
    lines.append(None)
    new_lines = []
    new_line = ''
    start = 0
    for end in range(len(lines)):
        if not lines[end]:
            margin = left_margin_size(lines[start:end])
            for line in lines[start:end]:
                counter = len(re.match(' *', line).group())
                if counter > margin:
                    if new_line:
                        new_lines.append(' ' * margin + new_line)
                        new_line = ''
                    indent = counter - margin
                else:
                    indent = 0
                for word in string.split(line):
                    if new_line:
                        if len(new_line) + 1 + len(word) > width:
                            new_lines.append(' ' * margin + new_line)
                            new_line = word
                        else:
                            new_line = new_line + ' ' + word
                    else:
                        new_line = ' ' * indent + word
                        indent = 0
            if new_line:
                new_lines.append(' ' * margin + new_line)
                new_line = ''
            if lines[end] is not None:
                new_lines.append('')
                start = end + 1
    return new_lines

.. @+node:ekr.20071103093725.23: *8* class Marker
class Marker:
    @others
.. @+node:ekr.20071103093725.24: *9* save_position

## Heuristic to simulate a marker while reformatting boxes.

def save_position(self, text, position, ignorable):
    """\
Given a TEXT and a POSITION in that text, save the adjusted position
by faking that all IGNORABLE characters before POSITION were removed.
"""
    ignore = {}
    for character in ' \t\r\n' + ignorable:
        ignore[character] = None
    counter = 0
    for character in text[:position]:
        if character in ignore:
            counter = counter + 1
    self.position = position - counter

.. @+node:ekr.20071103093725.25: *9* get_position
def get_position(self, text, ignorable, latest=0):
    """\
Given a TEXT, return the value that would yield the currently saved position,
if it was saved by `save_position' with IGNORABLE.  Unless the position lies
within a series of ignorable characters, LATEST has no effect in practice.
If LATEST is true, return the biggest possible value instead of the smallest.
"""
    ignore = {}
    for character in ' \t\r\n' + ignorable:
        ignore[character] = None
    counter = 0
    position = 0
    if latest:
        for character in text:
            if character in ignore:
                counter = counter + 1
            else:
                if position == self.position:
                    break
                position = position + 1
    elif self.position > 0:
        for character in text:
            if character in ignore:
                counter = counter + 1
            else:
                position = position + 1
                if position == self.position:
                    break
    return position + counter

.. @+node:ekr.20071103093725.26: *8* class Template
## Template processing.

class Template:
    @others
.. @+node:ekr.20071103093725.27: *9* __init_ (Template)_

def __init__(self, style, weight, lines):
    """\
Digest and register a single template.  The template is numbered STYLE,
has a parsing WEIGHT, and is described by one to three LINES.
STYLE should be used only once through all `declare_template' calls.

One of the lines should contain the substring `box' to represent the comment
to be boxed, and if three lines are given, `box' should appear in the middle
one.  Lines containing only spaces are implied as necessary before and after
the the `box' line, so we have three lines.

Normally, all three template lines should be of the same length.  If the first
line is shorter, it represents a start comment string to be bundled within the
first line of the comment text.  If the third line is shorter, it represents
an end comment string to be bundled at the end of the comment text, and
refilled with it.
"""
    assert style not in template_registry, \
           "Style %d defined more than once" % style
    self.style = style
    self.weight = weight
    # Make it exactly three lines, with `box' in the middle.
    start = string.find(lines[0], 'box')
    if start >= 0:
        line1 = None
        line2 = lines[0]
        if len(lines) > 1:
            line3 = lines[1]
        else:
            line3 = None
    else:
        start = string.find(lines[1], 'box')
        if start >= 0:
            line1 = lines[0]
            line2 = lines[1]
            if len(lines) > 2:
                line3 = lines[2]
            else:
                line3 = None
        else:
            assert 0, "Erroneous template for %d style" % style
    end = start + len('box')
    # Define a few booleans.
    self.merge_nw = line1 is not None and len(line1) < len(line2)
    self.merge_se = line3 is not None and len(line3) < len(line2)
    # Define strings at various cardinal directions.
    if line1 is None:
        self.nw = self.nn = self.ne = None
    elif self.merge_nw:
        self.nw = line1
        self.nn = self.ne = None
    else:
        if start > 0:
            self.nw = line1[:start]
        else:
            self.nw = None
        if line1[start] != ' ':
            self.nn = line1[start]
        else:
            self.nn = None
        if end < len(line1):
            self.ne = string.rstrip(line1[end:])
        else:
            self.ne = None
    if start > 0:
        self.ww = line2[:start]
    else:
        self.ww = None
    if end < len(line2):
        self.ee = line2[end:]
    else:
        self.ee = None
    if line3 is None:
        self.sw = self.ss = self.se = None
    elif self.merge_se:
        self.sw = self.ss = None
        self.se = string.rstrip(line3)
    else:
        if start > 0:
            self.sw = line3[:start]
        else:
            self.sw = None
        if line3[start] != ' ':
            self.ss = line3[start]
        else:
            self.ss = None
        if end < len(line3):
            self.se = string.rstrip(line3[end:])
        else:
            self.se = None
    # Define parsing regexps.
    if self.merge_nw:
        self.regexp1 = re.compile(' *' + regexp_quote(self.nw) + '.*$')
    elif self.nw and not self.nn and not self.ne:
        self.regexp1 = re.compile(' *' + regexp_quote(self.nw) + '$')
    elif self.nw or self.nn or self.ne:
        self.regexp1 = re.compile(
            ' *' + regexp_quote(self.nw) + regexp_ruler(self.nn)
            + regexp_quote(self.ne) + '$')
    else:
        self.regexp1 = None
    if self.ww or self.ee:
        self.regexp2 = re.compile(
            ' *' + regexp_quote(self.ww) + '.*'
            + regexp_quote(self.ee) + '$')
    else:
        self.regexp2 = None
    if self.merge_se:
        self.regexp3 = re.compile('.*' + regexp_quote(self.se) + '$')
    elif self.sw and not self.ss and not self.se:
        self.regexp3 = re.compile(' *' + regexp_quote(self.sw) + '$')
    elif self.sw or self.ss or self.se:
        self.regexp3 = re.compile(
            ' *' + regexp_quote(self.sw) + regexp_ruler(self.ss)
            + regexp_quote(self.se) + '$')
    else:
        self.regexp3 = None
    # Save results.
    template_registry[style] = self

.. @+node:ekr.20071103093725.28: *9* __cmp__
def __cmp__(self, other):
    return cmp(self.weight, other.weight)

.. @+node:ekr.20071103093725.29: *9* characters
def characters(self):
    """\
Return a string of characters which may be used to draw the box.
"""
    characters = ''
    for text in (self.nw, self.nn, self.ne,
                 self.ww, self.ee,
                 self.sw, self.ss, self.se):
        if text:
            for character in text:
                if character not in characters:
                    characters = characters + character
    return characters

.. @+node:ekr.20071103093725.30: *9* match
def match(self, lines):
    """\
Returns true if LINES exactly match this template.
"""
    start = 0
    end = len(lines)
    if self.regexp1 is not None:
        if start == end or not self.regexp1.match(lines[start]):
            return 0
        start = start + 1
    if self.regexp3 is not None:
        if end == 0 or not self.regexp3.match(lines[end-1]):
            return 0
        end = end - 1
    if self.regexp2 is not None:
        for line in lines[start:end]:
            if not self.regexp2.match(line):
                return 0
    return 1

.. @+node:ekr.20071103093725.31: *9* unbuild
def unbuild(self, lines):
    """\
Remove all comment marks from LINES, as hinted by this template.  Returns the
cleaned up set of lines, and the size of the left margin.
"""
    margin = left_margin_size(lines)
    # Remove box style marks.
    start = 0
    end = len(lines)
    if self.regexp1 is not None:
        lines[start] = unbuild_clean(lines[start], self.regexp1)
        start = start + 1
    if self.regexp3 is not None:
        lines[end-1] = unbuild_clean(lines[end-1], self.regexp3)
        end = end - 1
    if self.regexp2 is not None:
        for counter in range(start, end):
            lines[counter] = unbuild_clean(lines[counter], self.regexp2)
    # Remove the left side of the box after it turned into spaces.
    delta = left_margin_size(lines) - margin
    for counter in range(len(lines)):
        lines[counter] = lines[counter][delta:]
    # Remove leading and trailing white lines.
    start = 0
    end = len(lines)
    while start < end and lines[start] == '':
        start = start + 1
    while end > start and lines[end-1] == '':
        end = end - 1
    return lines[start:end], margin

.. @+node:ekr.20071103093725.32: *9* build
def build(self, lines, width, refill, margin):
    """\
Put LINES back into a boxed comment according to this template, after
having refilled them if REFILL.  The box should start at column MARGIN,
and the total size of each line should ideally not go over WIDTH.
"""
    # Merge a short end delimiter now, so it gets refilled with text.
    if self.merge_se:
        if lines:
            lines[-1] = lines[-1] + '  ' + self.se
        else:
            lines = [self.se]
    # Reduce WIDTH according to left and right inserts, then refill.
    if self.ww:
        width = width - len(self.ww)
    if self.ee:
        width = width - len(self.ee)
    if refill:
        lines = refill_lines(lines, width)
    # Reduce WIDTH further according to the current right margin,
    # and excluding the left margin.
    maximum = 0
    for line in lines:
        if line:
            if line[-1] in '.!?':
                length = len(line) + 1
            else:
                length = len(line)
            if length > maximum:
                maximum = length
    width = maximum - margin
    # Construct the top line.
    if self.merge_nw:
        lines[0] = ' ' * margin + self.nw + lines[0][margin:]
        start = 1
    elif self.nw or self.nn or self.ne:
        if self.nn:
            line = self.nn * width
        else:
            line = ' ' * width
        if self.nw:
            line = self.nw + line
        if self.ne:
            line = line + self.ne
        lines.insert(0, string.rstrip(' ' * margin + line))
        start = 1
    else:
        start = 0
    # Construct all middle lines.
    for counter in range(start, len(lines)):
        line = lines[counter][margin:]
        line = line + ' ' * (width - len(line))
        if self.ww:
            line = self.ww + line
        if self.ee:
            line = line + self.ee
        lines[counter] = string.rstrip(' ' * margin + line)
    # Construct the bottom line.
    if self.sw or self.ss or self.se and not self.merge_se:
        if self.ss:
            line = self.ss * width
        else:
            line = ' ' * width
        if self.sw:
            line = self.sw + line
        if self.se and not self.merge_se:
            line = line + self.se
        lines.append(string.rstrip(' ' * margin + line))
    return lines

.. @+node:ekr.20071103093725.33: *8* regexp_quote
def regexp_quote(text):
    """\
Return a regexp matching TEXT without its surrounding space, maybe
followed by spaces.  If STRING is nil, return the empty regexp.
Unless spaces, the text is nested within a regexp parenthetical group.
"""
    if text is None:
        return ''
    if text == ' ' * len(text):
        return ' *'
    return '(' + re.escape(text.strip() + ') *'

.. @+node:ekr.20071103093725.34: *8* regexp_ruler
def regexp_ruler(character):
    """\
Return a regexp matching two or more repetitions of CHARACTER, maybe
followed by spaces.  Is CHARACTER is nil, return the empty regexp.
Unless spaces, the ruler is nested within a regexp parenthetical group.
"""
    if character is None:
        return ''
    if character == ' ':
        return '  +'
    return '(' + re.escape(character + character) + '+) *'

.. @+node:ekr.20071103093725.35: *8* unbuild_clean
def unbuild_clean(line, regexp):
    """\
Return LINE with all parenthetical groups in REGEXP erased and replaced by an
equivalent number of spaces, except for trailing spaces, which get removed.
"""
    match = re.match(regexp, line)
    groups = match.groups()
    for counter in range(len(groups)):
        if groups[counter] is not None:
            start, end = match.span(1 + counter)
            line = line[:start] + ' ' * (end - start) + line[end:]
    return string.rstrip(line)

.. @+node:ekr.20071103093725.36: *8* make_generic
## Template data.

# Matcher functions for a comment start, indexed by numeric LANGUAGE.
language_matcher = []
for pattern in (r' *(/\*|//+|#+|;+|%+)',
                r'',            # 1
                r' */\*',       # 2
                r' *//+',       # 3
                r' *#+',        # 4
                r' *;+',        # 5
                r' *%+'):       # 6
    language_matcher.append(re.compile(pattern).match)

# Template objects, indexed by numeric style.
template_registry = {}

def make_generic(style, weight, lines):
    """\
Add various language digit to STYLE and generate one template per language,
all using the same WEIGHT.  Replace `?' in LINES accordingly.
"""
    for language, character in ((300, '/'),  # C++ style comments
                                (400, '#'),  # scripting languages
                                (500, ';'),  # LISP and assembler
                                (600, '%')): # TeX and PostScript
        new_style = language + style
        if 310 < new_style <= 319:
            # Disallow quality 10 with C++.
            continue
        new_lines = []
        for line in lines:
            new_lines.append(string.replace(line, '?', character))
        Template(new_style, weight, new_lines)

.. @+node:ekr.20071103093725.37: *8* << templates >>

make_generic(11, 115, ('? box',))

make_generic(12, 215, ('? box ?',
                       '? --- ?'))

make_generic(13, 315, ('? --- ?',
                       '? box ?',
                       '? --- ?'))

make_generic(14, 415, ('? box ?',
                       '???????'))

make_generic(15, 515, ('???????',
                       '? box ?',
                       '???????'))

make_generic(21, 125, ('?? box',))

make_generic(22, 225, ('?? box ??',
                       '?? --- ??'))

make_generic(23, 325, ('?? --- ??',
                       '?? box ??',
                       '?? --- ??'))

make_generic(24, 425, ('?? box ??',
                       '?????????'))

make_generic(25, 525, ('?????????',
                       '?? box ??',
                       '?????????'))

make_generic(31, 135, ('??? box',))

make_generic(32, 235, ('??? box ???',
                       '??? --- ???'))

make_generic(33, 335, ('??? --- ???',
                       '??? box ???',
                       '??? --- ???'))

make_generic(34, 435, ('??? box ???',
                       '???????????'))

make_generic(35, 535, ('???????????',
                       '??? box ???',
                       '???????????'))

make_generic(41, 145, ('???? box',))

make_generic(42, 245, ('???? box ????',
                       '???? --- ????'))

make_generic(43, 345, ('???? --- ????',
                       '???? box ????',
                       '???? --- ????'))

make_generic(44, 445, ('???? box ????',
                       '?????????????'))

make_generic(45, 545, ('?????????????',
                       '???? box ????',
                       '?????????????'))

# Textual (non programming) templates.

Template(111, 113, ('box',))

Template(112, 213, ('| box |',
                    '+-----+'))

Template(113, 313, ('+-----+',
                    '| box |',
                    '+-----+'))

Template(114, 413, ('| box |',
                    '*=====*'))

Template(115, 513, ('*=====*',
                    '| box |',
                    '*=====*'))

Template(121, 123, ('| box |',))

Template(122, 223, ('| box |',
                    '`-----\''))

Template(123, 323, ('.-----.',
                    '| box |',
                    '`-----\''))

Template(124, 423, ('| box |',
                    '\\=====/'))

Template(125, 523, ('/=====\\',
                    '| box |',
                    '\\=====/'))

Template(141, 143, ('| box ',))

Template(142, 243, ('* box *',
                    '*******'))

Template(143, 343, ('*******',
                    '* box *',
                    '*******'))

Template(144, 443, ('X box X',
                    'XXXXXXX'))

Template(145, 543, ('XXXXXXX',
                    'X box X',
                    'XXXXXXX'))
# C language templates.

Template(211, 118, ('/* box */',))

Template(212, 218, ('/* box */',
                    '/* --- */'))

Template(213, 318, ('/* --- */',
                    '/* box */',
                    '/* --- */'))

Template(214, 418, ('/* box */',
                    '/* === */'))

Template(215, 518, ('/* === */',
                    '/* box */',
                    '/* === */'))

Template(221, 128, ('/* ',
                    '   box',
                    '*/'))

Template(222, 228, ('/*    .',
                    '| box |',
                    '`----*/'))

Template(223, 328, ('/*----.',
                    '| box |',
                    '`----*/'))

Template(224, 428, ('/*    \\',
                    '| box |',
                    '\\====*/'))

Template(225, 528, ('/*====\\',
                    '| box |',
                    '\\====*/'))

Template(231, 138, ('/*    ',
                    ' | box',
                    ' */   '))

Template(232, 238, ('/*        ',
                    ' | box | ',
                    ' *-----*/'))

Template(233, 338, ('/*-----* ',
                    ' | box | ',
                    ' *-----*/'))

Template(234, 438, ('/* box */',
                    '/*-----*/'))

Template(235, 538, ('/*-----*/',
                    '/* box */',
                    '/*-----*/'))

Template(241, 148, ('/*    ',
                    ' * box',
                    ' */   '))

Template(242, 248, ('/*     * ',
                    ' * box * ',
                    ' *******/'))

Template(243, 348, ('/******* ',
                    ' * box * ',
                    ' *******/'))

Template(244, 448, ('/* box */',
                    '/*******/'))

Template(245, 548, ('/*******/',
                    '/* box */',
                    '/*******/'))

Template(251, 158, ('/* ',
                    ' * box',
                    ' */   '))

.. @+node:ekr.20130814140750.17191: *4* IPython notes
.. @+node:ekr.20130814140750.17192: *5* Ipython now has cell magic
From: Alia K <alia_khouri@yahoo.com>

Fernando Perez of IPython fame just announced recently that IPython
trunk has the cell magic functionality that was discussed earlier
[http://groups.google.com/group/leo-editor/browse_thread/thread/
7d910a68072dda1/b14e84fc3cfddbf6?lnk=gst&q=ipython#b14e84fc3cfddbf6].
As this could work very nicely with leo as an ipython editor, I will
include his text verbatim here:

===== Fernando

I'm excited to report that we now have cell magics in IPython... PR
1732 [1] has just been merged [2], which implements the design
discussed in IPEP 1 [3]. This is probably one of the largest PRs we've
had so far, with over 100 commits, over 100 comments and a diff that's
almost 11000 lines long (a lot of it moving code around, obviously
it's not all new code).  But it brings two very important thigns:

1) a refactor of the magic system to finally remove the old mixin
class we'd had since the very first days of IPython in 2001.  This is
a cleanup I've been wanting to do for over 10 years!  The new setup
makes the magic system have  a very clean api, that is easy to use
both for the implementation of core features and for users to create
their own magics.

2) the new concept of cell magics: these are magics that get not only
the line they're on, but the entire cell body as well.  And while
these are most naturally used in the notebook, as you would expect
we've built them at the core of IPython, so you can use them with all
the clients (terminal, qt console, notebook).  For example, this is a
Cython magic that Brian just prototyped out (we'll have a production
version of it soon included).  Note that this was copied *from a
regular text terminal*, not from the notebook:

In [3]: from IPython.core.magic import register_line_cell_magic

In [4]: @register_line_cell_magic
   ...: def cython(line, cell):
   ...:     """Compile and import a cell as a .pyx file."""
   ...:     import sys
   ...:     from importlib import import_module
   ...:     module = line.strip()
   ...:     fname = module + '.pyx'
   ...:     with open(fname,'w') as f:
   ...:         f.write(cell)
   ...:     if 'pyximport' not in sys.modules:
   ...:         import pyximport
   ...:         pyximport.install(reload_support=True)
   ...:     globals()[module] = import_module(module)
   ...:

In [5]: %%cython bam
   ...: def f(x):
   ...:     return 2.0*x
   ...:

In [6]: bam.f(10)
Out[6]: 20.0

In a similar spirit, Jonathan Taylor recently created one to call R
transparently in the notebook:

https://github.com/jonathan-taylor/Rmagic

This one hasn't been fully updated to the final API, but the core code
is there and now it should be a trivial matter to update it.


I want to thank everyone who pitched in with ideas during the
discussion and review period, and I hope you'll all enjoy this and
come up with great ways to use the system.  For now, you can see how
the system works by playing with %%timeit and %%prun, the only two
builtins that I extended to work also as cell magics.

For more details, see the documentation where we've added also a long
new section with details and examples of how to create your own [4].

Cheers,

f

[1] https://github.com/ipython/ipython/pull/1732
[2] https://github.com/ipython/ipython/commit/61eb2ffeebb91a94fe9befe2c30e7839781ddc52
[2] https://github.com/ipython/ipython/issues/1611
[3] http://ipython.org/ipython-doc/dev/interactive/reference.html#magic-command-system

.. @+node:ekr.20130814140750.17193: *5* IPython code notes
Investigate how IPython hijacks event loops
http://groups.google.com/group/leo-editor/browse_thread/thread/e1dc6439bf8b17f9

pyos_inputhook is relevant

IPython lib.inputhook
http://ipython.org/ipython-doc/stable/api/generated/IPython.lib.inputhook.html

* IPython seems to require Python 2.x.
* I can run IPython from either C:\prog\ipython-0.12 or from python\lib\site-packages

From C:\prog\ipython-0.12\IPython\scripts

#!/usr/bin/env python
"""Terminal-based IPython entry point.
"""

from IPython.frontend.terminal.ipapp import launch_new_instance

launch_new_instance()

Here is ipapi.get::

@language python

    def get():
        """Get the global InteractiveShell instance."""
        from IPython.core.interactiveshell import InteractiveShell
        return InteractiveShell.instance()
        
See also:
    
http://ipython.org/ipython-doc/rel-0.12/api/index.html
http://ipython.org/ipython-doc/rel-0.12/api/generated/IPython.core.interactiveshell.html


.. @+node:ekr.20031218072017.392: *4* Python Notes...
.. @+node:ekr.20031218072017.398: *5* How to call any Python method from the C API
In general, everything you can do in Python is accessible through the C API.

    lines = block.split('\n');

> That will be

    lines = PyObject_CallMethod(block, "split", "s", "\n");
.. @+node:ekr.20031218072017.399: *5* How to run Python programs easily on NT,2K,XP
.. @+node:ekr.20031218072017.400: *6* setting the PATHEXT env var
It is worth noting that NT, Win2K and XP all have an alternative which is
to add .PY to the PATHEXT environment variable. Then you can run any .PY
file directly just by typing the name of the script without the extension. 

e.g.
C:\>set PATHEXT=.COM;.EXE;.BAT;.CMD

C:\>set PATH=%PATH%;c:\python22\tools\Scripts

C:\>google
'google' is not recognized as an internal or external command,
operable program or batch file.

C:\>set PATHEXT=.COM;.EXE;.BAT;.CMD;.PY

C:\>google
Usage: C:\python22\tools\Scripts\google.py querystring

C:\>
.. @+node:ekr.20031218072017.401: *6* Yet another Python .bat wrapper
>> It has a header of just one line. All the ugly stuff is at the end.
>>
>> -------------------------------------------------------------------
>> goto ="python"
>>
>> # Python code goes here
>>
>> ''' hybrid python/batch footer:
>> @:="python"
>> @python.exe %0 %1 %2 %3 %4 %5 %6 %7 %8 %9
>> @if errorlevel 9009 echo Python may be downloaded from
>www.python.org/download
>> @rem '''
>> -------------------------------------------------------------------
>>
>>         Oren
>>
>

It's for running python scripts on windows, without having to type:

[<path to python>\]python[.exe] <scriptname> [<arguments>*]

and almost takes the place of the "shabang" line at the top of *nix
scripts.

.. @+node:ekr.20080110082845: *4* pyxides: code completion
Python code completion module

From: "Tal Einat" <talei...@gmail.com>
Date: Wed, 6 Jun 2007 20:57:18 +0300

I've been developing IDLE over the past 2 years or so. Even before
that, I helped a friend of mine, Noam Raphael, write IDLE's
auto-completion, which is included in recent versions of IDLE.

Noam wrote the original completion code from scratch, and AFAIK every
Python IDE which features code completion has done the same. Surely
there is -some- functionality which could be useful cross-IDE?
Retrieving possible completions from the namespace, for example. And
we should be learning from each-others' ideas and experiences.

So how about we design a generic Python completion module, that
each IDE could extend, and use for the completion logic?



From: "Ali Afshar" <aafs...@gmail.com>
Date: Wed, 6 Jun 2007 19:06:01 +0100

I am very keen for this. I will help where it is required. PIDA
currently has no code completion (outside what vim/emacs provide),



From: "phil jones" <inters...@gmail.com>
Date: Wed, 6 Jun 2007 11:07:33 -0700

What functions would we ask for a code completion module?

Presumably recognition of the beginnings of
- a) python keywords
- b) classes and functions defined earlier in this file?
- c) in scope variables?

As python is dynamically typed, I guess we can't expect to know the
names of methods of objects?



From: "Ali Afshar" <aafs...@gmail.com>
Date: Wed, 6 Jun 2007 19:13:10 +0100

> Presumably recognition of the beginnings of
> - a) python keywords
> - b) classes and functions defined earlier in this file?
> - c) in scope variables?

does c) include: d) imported modules



From: Nicolas Chauvat <nicolas.chau...@logilab.fr>
Date: Wed, 6 Jun 2007 20:17:30 +0200

> >Presumably recognition of the beginnings of
> >- a) python keywords
> >- b) classes and functions defined earlier in this file?
> >- c) in scope variables?

> does c) include: d) imported modules

For code-completion, I suppose astng[1] could be useful.

1: http://www.logilab.org/project/eid/856


From: Stani's Python Editor <spe.stani...@gmail.com>
Date: Wed, 06 Jun 2007 20:48:41 +0200

A good point. I think we all have been thinking about this. Important
issues for the design is the extraction method and the sources.

*the method*
Importing is a lazy, but accurate way of importing, but is security wise
not such a good idea. Parsing throught an AST compiler is better,
however more difficult. Here are two options.

From version 2.5 the standard Python compiler converts internally the
source code to an abstract syntax tree (AST) before producing the
bytecode. So probably that is a good way to go as every python
distribution has this battery included.

As Nicolas suggested earlier on this mailing list, there is another
option: the AST compiler in python or PyPy:

On Mar 14 2006, 12:16 am, Nicolas Chauvat <nicolas.chau...@logilab.fr>
wrote:

> > WingIDE use anASTgenerator written in C (but cross-platform),
> > lightningly quick, and open sourced. This could be a potential
> > starting point.

> > Additionally isn't Python2.5 planned to have a C-written compiler?

> PyPy also produced an improved parser/compiler.

> http://codespeak.net/pypy/dist/pypy/doc/index.html
> http://codespeak.net/pypy/dist/pypy/module/recparser/

But if it could be done with the standard one it is one dependency less.

*the sources*
In the design we could define first the sources:
1 external imported modules from the pythonpath
2 local modules relative to the current file or context dependent
(Blender, Gimp, ...)
3 inner code

For 1:
It might be a good idea to have a function which scans all the modules
from the pythonpath or one specific module to cache all autocompletion
and calltip information of all classes, methods and doc strings. Why?
Modules in the pythonpath don't change so often. With some criteria
(file name, time stamp, size, ...) you could check if updates are
necessary at startup. Having a readymade 'database' (could be python
dictionary or sqlite database) for autocompletion/call tips would speed
up things (and is also more secure if you are importing rather than
parsing. For example trying to provide gtk autocompletion in a wxPython
by importing is problematic).

For 2:
Here you load the parser on demand. Autocompletion/calltip information
can be added to the database.

For 3:
A different kind of parser needs to be used here as per definition code
you edit contains errors while typing. External modules are retrieved
from 1 and 2, for internal code you can scan all the words and add them
to the autocomplete database. As a refinement you can give special
attention to 'self'. Also for calltips you can inherit when there are
assignments, eg
frame = Frame()
than frame inherits autocomplete & calltip information from Frame.

So autocompletion & calltips deals with two steps: extraction and
'database'. If someone has a good parser already, we could use it.
Otherwise we can define an API for the extraction and maybe lazily
implement it first with importing and concentrate first on the
'database'. When the database is ready we can implement the parsing. You
could also implement the parsing first, but than it takes longer before
you have results. Of course the library is GUI independent, it only
works with strings or lists.

What concerns SPE, it uses importing for autocompletion (1+2) and does
internal code analysis for local code (however without the inheriting).

Tal, how does IDLE's autocompletion works?

Stani



From: Stani's Python Editor <spe.stani...@gmail.com>
Date: Wed, 06 Jun 2007 20:53:10 +0200

Nicolas Chauvat wrote:
> On Wed, Jun 06, 2007 at 07:13:10PM +0100, Ali Afshar wrote:
>>> Presumably recognition of the beginnings of
>>> - a) python keywords
>>> - b) classes and functions defined earlier in this file?
>>> - c) in scope variables?
>> does c) include: d) imported modules

> For code-completion, I suppose astng[1] could be useful.

> 1: http://www.logilab.org/project/eid/856

How dependent/independent is this from the standard AST compiler or
PyPy? Is it more IDE friendly? Is it based on it or a total independent
implementation?



From: "Ali Afshar" <aafs...@gmail.com>
Date: Wed, 6 Jun 2007 19:59:13 +0100

> A good point. I think we all have been thinking about this. Important
> issues for the design is the extraction method and the sources.

> *the method*
> Importing is a lazy, but accurate way of importing, but is security wise
> not such a good idea. Parsing throught an AST compiler is better,
> however more difficult. Here are two options.

> From version 2.5 the standard Python compiler converts internally the
> source code to an abstract syntax tree (AST) before producing the
> bytecode. So probably that is a good way to go as every python
> distribution has this battery included.

> As Nicolas suggested earlier on this mailing list, there is another
> option: the AST compiler in python or PyPy:

What concerns me about these is whether they would work in a module
which has a syntax error.

I believe Wing's compiler bit of their code completion is open source.
I remember having seen the code.



From: Stani <spe.stani...@gmail.com>
Date: Wed, 06 Jun 2007 12:08:00 -0700

> What concerns me about these is whether they would work in a module
> which has a syntax error.

> I believe Wing's compiler bit of their code completion is open source.
> I remember having seen the code.

It is indeed, but is implemented in C, which means an extra dependency
and not a 100% python solution. Normally modules (especially in the
pythonpath) which you import don't have syntax errors. Maybe logilabs
implementation handles syntax errors well as it is developed for
PyLint. Nicolas?



From: "Tal Einat" <talei...@gmail.com>
Date: Wed, 6 Jun 2007 22:34:41 +0300

> As python is dynamically typed, I guess we can't expect to know the
> names of methods of objects?

Well, the dir() builtin does just that, though there can be attributes
which won't be included therein. However, the builtin dir() can be
overridden... and ignoring it can break libraries like RPyC which
define a custom dir() function just for this purpose.

This issue has already been run in to by RPyC (an Python RPC lib). The
main developr went ahead and suggested adding a __dir__ method which
will return a list of attributes, and IIRC he has already implemented
a patch for this, and it will likely enter Python2.6.

Until then, I guess we're going to have to rely on dir for this.



From: "Josiah Carlson" <josiah.carl...@gmail.com>
Date: Wed, 6 Jun 2007 12:42:01 -0700

For reference, PyPE auto-parses source code in the background, generating
(among other things) a function/class/method hierarchy.  Its autocomplete
generally sticks to global functions and keywords, but when doing
self.method lookups, it checks the current source code line, looks up in its
index of classes/methods, and trims the results based on known methods in
the current class in the current source file.

It certainly isn't complete (it should try to check base classes of the
class in the same file, it could certainly pay attention to names assigned
in the current scope, the global scope, imports, types of objects as per
WingIDE's assert isinstance(obj, type), etc.), but it also makes the
computation fairly straightforward, fast, and only in reference to the
current document.



From: "Tal Einat" <talei...@gmail.com>
Date: Wed, 6 Jun 2007 22:52:08 +0300

> Tal, how does IDLE's autocompletion works?

Much like Stani said, since Python is interpreted, collection of
possible completions splits into two methods:
1) source code analysis
2) dynamic introspection

Of course, we could do either or a combination of both.

IDLE just uses introspection: since IDLE always has a python shell
running, it just completes according to the shell's state (plus
built-in keywords and modules). This is a very simple method,
obviously lacking. It does allow the user some control of the
completion, though - just import whatever you want to be completable
in the shell. However, introspection is all that is needed in a Python
shell, which is the major reason this is the method used in IDLE.



From: Nicolas Chauvat <nicolas.chau...@logilab.fr>
Date: Wed, 6 Jun 2007 23:59:32 +0200


> How dependent/independent is this from the standard AST compiler or
> PyPy? Is it more IDE friendly? Is it based on it or a total independent
> implementation?

It is independent from PyPy.

The above web page says:

"""
Python Abstract Syntax Tree New Generation

The aim of this module is to provide a common base representation of
python source code for projects such as pychecker, pyreverse,
pylint... Well, actually the development of this library is essentialy
governed by pylint's needs.

It extends class defined in the compiler.ast [1] module with some
additional methods and attributes. Instance attributes are added by a
builder object, which can either generate extended ast (let's call
them astng ;) by visiting an existant ast tree or by inspecting living
object. Methods are added by monkey patching ast classes.Python
Abstract Syntax Tree New Generation

The aim of this module is to provide a common base representation of
python source code for projects such as pychecker, pyreverse,
pylint... Well, actually the development of this library is essentialy
governed by pylint's needs.

It extends class defined in the compiler.ast [1] module with some
additional methods and attributes. Instance attributes are added by a
builder object, which can either generate extended ast (let's call
them astng ;) by visiting an existant ast tree or by inspecting living
object. Methods are added by monkey patching ast classes.
"""

From: "Sylvain Thénault" <thena...@gmail.com>
Date: Wed, 13 Jun 2007 10:51:04 +0200

> Please let me involve Sylvain in the discussion. As the main author of
> pylint and astng, he will provide better answers.

well logilab-astng is basically a big monkey patching of the compiler
package from the stdlib, so you can't get an astng representation from a
module with syntax errors in. However inference and most others
navigation methods (which are basically the value added by astng) are
"syntax error resilient" : if a dependency module (direct or indirect)
contains a syntax error, you don't get any exception, though since some
information is missing you can miss some results you'ld get if the
faulting module were parseable.



From: "Tal Einat" <talei...@gmail.com>
Date: Tue, 31 Jul 2007 10:33:33 +0300

Since astng already does some inference (which we definitely want!)
and is based on the standard Python AST compiler, it sounds like our
#1 candidate. I think we should give the code a serious once-over and
see how well it fits our requirements, and if it can be adapted to
better handle errors. Any volunteers?

Also, has anyone used astng for completion, calltips, or something
similar? Or the standard AST compiler, for that matter?



From: "Tal Einat" <talei...@gmail.com>
Date: Tue, 31 Jul 2007 10:40:11 +0300

How does PyPE parse code? Home-rolled, standard AST compiler, something else?

It seems to me we should try to come up with an algorithm for parsing,
before getting to the code. All of the details you mentioned -
noticing assignments, using base-class methods, etc. - could be better
defined and organized this way. Perhaps we could brainstorm on this in
a wiki?



From: "Tal Einat" <talei...@gmail.com>
Date: Tue, 31 Jul 2007 11:38:40 +0300

Sorry for being away for such a long time. I hope we can get this
conversation rolling again, and get started with the actual work.

I'll try to sum up what has been said so far, and how I see things.

== Top Priorities ==
* Can we implement a parser based on the standard Python AST compiler
(or astng)? For example, can syntax errors be handled well?
* Is importing reasonable security-wise? If not, can it be made secure?

== General issues ==
* Do we aim for just completion, or also calltips? Perhaps also other
meta-data, e.g. place defined, source code, ... (see IPython's '??')
* Dependencies - do we want to allow C-extensions, or are we going for
a Python-only solution? (IDLE would only use such a Python-only tool.)
It seems that we want to pre-process most of the data in the
background, so I don't see why we would want to do this in C for
efficiency reasons.

== Completion sources ==
1) Importing "external" modules
2) Importing/Parsing "local" modules
3) Parsing the current file
4) Using objects/modules from the shell (e.g. IDLE has both editor
windows and a Python shell)

== Importing ==
* Stani mentioned that importing is problematic from a security point
of view. What are the security issues? Are they really an issue for an
IDE? If so, perhaps we could overcome this by importing in some kind
of "sandbox"?
* What are the pros and cons of Importing vs. Parsing?
* If importing is always preferable to parsing unless there's a syntax
error, perhaps try to import and parse on failure?

== Parsing ==
* This is going to be the most complex method - I think we should have
a general idea of how this should work before starting an
implementation. I suggest hashing ideas out on a wiki, since there a
lot of details to consider.
* Can a parser based on the standard AST compiler (or astng) work? Is
there a way to deal with errors? (HIGH PRIORITY!)
* There are other existing, open-source implementations out there -
WingIDE, PyPE have been mentioned. Any others? We should collect these
so we can use the code for learning, and perhaps direct use (if
possible license-wise).

== Shell ==
This is relatively straight-forward - just use dir(). This should be
optional, for use by IDEs which have a shell (support multiple
shells?).

Some known issues from IDLE and PyCrust:
* Handle object proxies such as RPC proxies (e.g. RPyC)
* Handle ZODB "ghost" objects
* Watch out for circular references
* Watch out for objects with special __getattr__/__hasattr__
implementations (for example xmlrpc, soap)

== Persistence ==
* Stani mentioned a 'database'. I feel Sqlite should be at most
optional, to reduce dependencies.
* Do we really want to have the data persistent (between IDE
sessiosns)? If so, we need to support simultaneous instances of the
IDE so they don't corrupt the data. Any other issues? (I have a
feeling this would better be left for later stages of development.)



From: "Tal Einat" <talei...@gmail.com>
Date: Tue, 31 Jul 2007 12:22:59 +0300

One more note: We should distinguish between completion in an editor
and completion in a shell. The conversation up until now has focused
on editors, which is reasonable since that is the problematic scene. I
think a generic Python completion library should support completion in
both contexts, especially if it uses can use a shell's namespace for
completion in the editor.



From: "Ali Afshar" <aafs...@gmail.com>
Date: Tue, 31 Jul 2007 11:20:19 +0100

I have just implemented a completion mockup using Rope (which is a
refactoring library). It works quite nicely, and definitely worth a
look.

http://rope.sourceforge.net/

It even achieves this kind of completion:

class Banana(object):
    def do_something(self):
         return

def foo():
    return [Banana(), Banana()]

foo()[0].<complete> includes do_something

Which seems pretty impressive to me.



From: "Tal Einat" <talei...@gmail.com>
Date: Tue, 31 Jul 2007 20:12:50 +0300

Wow, Rope does look very impressive! A quick look at the code tells me
that a lot of work has been invested in it.

So we have one existing Python-only solution. We should evaluate it -
see what it can and can't do, and perhaps take a look at the overall
design.

I'm CC-ing Rope's developer, Ali. Hopefully Ali can help us quickly
understand Rope's code analysis capabilities.

Ali, could you elaborate a bit on what kinds of completion Rope can
do, and the methods it uses? We would especially like to know how your
static and dynamic inference work, what they can accomplish, and what
their limitations are.



From: "Ali Afshar" <aafs...@gmail.com>
Date: Tue, 31 Jul 2007 19:45:15 +0100

> Ali, could you elaborate a bit on what kinds of completion Rope can
> do, and the methods it uses? We would especially like to know how your
> static and dynamic inference work, what they can accomplish, and what
> their limitations are.

Well, I haven't really looked at the code. But I can tell you this:

from rope.ide.codeassist import PythonCodeAssist
from rope.base.project import Project
for compl in PythonCodeAssist(Project(package_root)).assist(buffer,
offset).completions:
    print compl

And that is as far as I really got. I expect to get a better look at
it later in the week though...


From: "Josiah Carlson" <josiah.carl...@gmail.com>
Date: Wed, 1 Aug 2007 00:26:14 -0700

> How does PyPE parse code? Home-rolled, standard AST compiler, something else?

The compiler for syntactically correct Python, a line-based compiler
for broken Python.  TO generate a method list for self.methods, using
the current line number, I discover the enclosing class, check the
listing of methods for that class (generated by the compiler or
line-based parsers), and return a valid list for the specified prefix.
 It doesn't walk the inheritance tree, it doesn't do imports, etc.

> It seems to me we should try to come up with an algorithm for parsing,
> before getting to the code. All of the details you mentioned -
> noticing assignments, using base-class methods, etc. - could be better
> defined and organized this way. Perhaps we could brainstorm on this in
> a wiki?

A wiki would be fine, the one for this mailing list would likely be
best (if it is still up and working).  Then again, Rope looks quite
nifty.  I may have to borrow some of that source ;)


Discussion subject changed to "Fwd: Python code completion module" by Tal Einat

From: Ali Gholami Rudi <aligr...@gmail.com>
Date: Aug 1, 2007 5:50 PM

First of all I should note that rope's main goal was being a
refactoring tool and a refactoring tool needs to know a lot about
python modules.  `rope.base` package provides information about python
modules.

Actually what ropeide provides as auto-completion is defined in
`rope.ide.codeassist` module.  This module almost does nothing but use
`rope.base`.  Since `rope.ide` package is not included in the rope
library (which has been separated from ropeide since 0.6m4) it lacks
good documentation and the API might not be easy to use (most of it is
written in the first months of rope's birth).

> ..., could you elaborate a bit on what kinds of completion Rope can
> do, ...

I don't know what to say here.  Well, actually it tries to use the
source code as much as possible and infer things from it.  So I can
say that it can complete any obvious thing that can be inferred by a
human.  Like this is the first parameter of a method and after dots
its attributes can appear or these modules are imported so their names
and contents are available or this is an instance of some known type
and we know its attributes and ... .  Try ropeide (it uses emacs-like
keybinding, C-/ for completion; see ~/.rope if you want to change
that); it completes common cases (and sometimes completes things you
don't expect it to!).

> ..., and the methods it uses?

Rope analyzes python source code and AST.  Rope used to use the
`compiler` module till 0.5 and now it uses `_ast` module.

> We would especially like to know how your
> static and dynamic inference work, what they can accomplish

There are a few examples in docs/overview.txt.  Unit-test modules like
`ropetest.base.objectinfertest` and `advanced_oi_test` might help,
too.  Also have a look at `rope.base.oi.__init__` pydoc for an
overview of how they work; (I'm afraid it is a bit out of date and
carelessly written.)  The idea behind rope's object inference is to
guess what references (names in source-code) hold.  They collect
information about code when they can and use them later.

>..., and what their limitations are.

Many things in rope are approximations that might be exact if some
conditions hold.  For instance rope might assume that every normal
reference in module scope holds only one kind of object.  Apart from
these assumptions both SOI and DOI have their own disadvantages; For
instance SOI fails when dynamic code is evaluated while DOI does not.
Or DOI is slower than SOI.  (Well, after recent enhancements to rope's
SOI I rarely use DOI).

I tried to answer as short as possible.  If there are questions on
specific parts of rope, I'll be happy to answer.

By the way, I tried to reply this mail to the group, but it seems that
your group requires subscription for posting, so I've sent it to you,
instead.
.. @+node:ekr.20131001045038.19027: *4* Quotes from "Why Leo isn't more popular"
@language rest
.. @+node:ekr.20131001045038.19028: *5* Fidel
When I say Learn Leo, I don't mean Learn or understand the code, I mean
learn what you can do with it.

My true feeling is that Leo is like an infinite ground, where amazing
things can be built. Some of Leo users are already doing incredible things
with it. But when you are new to Leo, you first have to learn the physics
of this new world (Leo Code), and then develop your own engineering (Useful
outlines structures) for building your "information cities"...

Possibly, experienced leo users dont share this view since they learned Leo
physics and engineering long time ago, but those look like a big wall when
starting with Leo.

So Leo files with samples would be like delivering this infinite ground
with some pre-built "information cities" so people can begin using them
right away. Ideally, those leo files should guide the user on how to build
new structures, and in the end, be able to lead him to generate new
structures.

My (again, personal) feeling is that the current policy right now is: Let
him read the code to understand how it works. So only the users willing to
go through the (1) Learn the physics then (2) develop your own way to
develop "information cities" can actually use leo and use its potential.

Since most of the internet users are the ones who need the cities already
built for them to use software, thats the chunk we are loosing.

On the other hand, coming to why programmers (who have the skills to go
through Leo learning process) dont come and use it, its the same story:

Before coming into Leo, what I was doing was actually search for an IDE
that I liked. I went through several, and in the ones I "liked" the most,
it would take you around two hours learning for having a folder with your
scripts and being able to edit them. With Leo, you first have to study and
understand what are directives, how do they work, how everything interacts
in the tree, etc. It takes more than two hours. I would say that it can be
measured in weeks or months until you really know what you are doing.

And here is where the "information cities" come again. If Leo were provided
with an interactive Leo file, which would guide you into: - Importing your
scripts to Leo - Clearly create folders and move/manage your files there -
Suggest directives for those files. - Etc.

That Leo file would save the new user many many hours and he would
instantly start using leo, so we would help him skip the first wall and
begin to get interested.

So then again, this is why I still see that Leo default workbook should be
dispatched with a Leo cheatsheet open by default, so new users begin to
play with it as soon as they open Leo.
.. @+node:ekr.20131001045038.19029: *5* Jacob Peck
Leo represents a completely new paradigm for editing, programming, and
interacting with data. Well, not completely new, but for many the surface
level seems extremely foreign.

Leo is primarily a tool aimed at python...the paradigm it exposes to the
user is far more like Lisp (homoiconicity via the Node and Tree structure)
and Smalltalk (.leo files save *most* of your programming environment's
state, including data). This is a huge cognitive leap for people only
accustomed to more imperative languages like Python...Leo asks a lot of

-----

​The PIM aspect was what initially led me to Leo. Wikipedia pointed me in
Leo's direction while searching for some good open-source PIM tools. I
tried Chandler but it didn't do what I wanted. Leo did.

.. @+node:ekr.20131001045038.19030: *5* Kent
Re: the original question "Why isn't Leo more popular" I can ask myself
"Why don't I adopt tools which promise a better way to work?" I do a lot of
grazing, and encounter very many such tools.

A tool which does everything doesn't grab me, I don't need to do
everything. A tool which, in it's few moments of my attention, describes a
better way to do something I need, gets additional moments. If the tool
salesperson mentions a task I encounter and a better way to do it, with a
learning curve which justifies the benefit ... I will settle in for a
while.

Easier, more powerful boilerplates / templates / completion is the kind of
bait I'll bite on, there's potential for decent return on investment.
.. @+node:ekr.20131001045038.19031: *5* Fidel
With "bread crumbs" I refer to...the interactive tutorial creator I have
talked so much about. My aim with this tool is that anything you do,
instantly becomes a tutorial, that can be done again and played forward and
backwards. Basically, your actions become an outline, and other users can
"play them" again. This should mean that making a tutorial will take as
much time as it takes for you to perform the action originally.
.. @+node:ekr.20131001045038.19032: *5* dufriz
As a newbie, I agree. At least, that is the perception, which is
discouraging. I believe it also has to do with the lack of learning
material. The only manual is not very newbie-friendly. It contains a lot of
cross-references to advanced internal commands (@xyz stuff) which put off
newcomers, especially where they are thrown in altogether and may look
scary.

Currently the learning curve does nor _appear_ to be gentle at all. But it
does not necessarily have to be like this. Even if Leo is indeed very
complex and very advanced, I am sure that at least a subset of Leo
functionality can be selected and presented in a nicer way, to attract more
users and help newbies.

Why can't we have some more accessible tutorial, with gentle, self
contained lessons (maybe webcasts, or slides, or simple HTML pages) which
_focus on the basic functionality_? Or perhaps, a two-tiered series of such
lessons, i.e. basic and advanced levels. I think this would be very
helpful.

-----

Another point: Leo is commonly advertised as a programming editor, but it
should be made more clear that it is more than that.

I believe you would attract more users if you also mentioned its usefulness
for general-purpose editing and PIM functionality. Of course, that may be
implicit, but I think you should be putting some more emphasis on this
aspect.

----- 

For me, the trajectory was KeyNote --> MyBase --> NotecasePro/RightNote -->
Leo (!!)

Leo is the best, because it is fully customizable, and gives you seemingly
infinite power. Only, it takes a lot to learn how to use that power.
.. @+node:ekr.20131001045038.19034: *5* dufriz
Consider that Leo is actively developed, and has a responsive community.
When I requested the support for Rich Text, a couple of months ago, it was
promptly implemented in a matter of days.

As for the power, of course Leo can do whatever other PIMs can do, because
it is extensible by Python. The question is whether enough people are
interested in Leo as a PIM, so as to have enough user contributions. I am
still at the beginning, so I have no idea how difficult it may be,
technically, to impement all the PIM features that we need. But if we
succeed in getting enough people interested, I am sure that nothing is
beyond our reach.

As far as PIM features go the next step, I believe, is implementing a good
tagging system.
.. @+node:ekr.20131003040744.18223: *5* dufriz **
As a newbie who still has zero knowledge of Leo's technical details, I can
say: I have absolutely no problem in conceptualizing what Leo is and what
it can do, and probably I also get the Leo's aha -- the difficulty is in
getting into the actual technical aspects of the program. Again, in the end
this all boils down to newbie-friendly tutorials and learning materials...
Something like Leo for dummies would be greatly appreciated.

.. @+node:ekr.20131001045038.19035: *5* Terry
I think a lot of the problem is defining what Leo is. In some ways I think
it's Python's Squeak (Squeak http://www.squeak.org/Screenshots/ being a
SmallTalk environment where everything's an object).

If all you want is an off the shelf turn key text editor with perhaps
completion and syntax highlighting as its most advanced features, then
there are several out there that are probably going to appeal more than
Leo.

If you're looking for an off the shelf turn key outliner, without the
intent to use scripts etc., then again there are lots of alternatives, some
of which probably seem more intuitive / simpler / even more feature rich
than Leo. (The feature rich part is probably misleading, but they may seem
that way initially).

If your looking for an *environment* which includes a good editor and
outliner and is completely scriptable / interactive / "live-code" in
Python, then Leo wins hands down. Of course, it's basically alone in this
field, as far as I know, but I'm sure it would do well even if it
wasn't...I guess Emacs is sort of an environment like this, only in Lisp
with a prehistoric GUI system.

Sometimes I've wondered why Leo seems to appeal to people who aren't
comfortable in Python, I think now it's because they still recognize the
value of an *environment*, and there's just not much in this niche. So to
me it's more why don't more people want an environment like this...

My feeling, talking generally and not about Leo in particular, is that
while not everyone needs to be a "programmer", everyone who uses computers
would benefit from being able to interact with them with more of the
flexibility and efficiency that comes with even relatively simple
"scripting", and less of the limitations that come with the "buy our latest
product and you'll be able to do anything you want with a click" view
pushed by other suppliers of computing environments.
.. @+node:ekr.20131001045038.19036: *5* Miles Fidelman (improve wiki)
In that regard, Leo is barely mentioned on WikiPedia - it has a simple
descriptive page, but on the list of text editors it's listed, but there's
no information about o/s support or features.

For marketing, packaging, features, documentation, tutorials,
extensions,... - do a side by side comparison of Vim and Leo - and the
answer(s) to "why leo isn't more popular" leap out at you.
.. @+node:ekr.20131001045038.19038: *5* Chris George
I came to Leo because it is the only outliner for Linux that supports
cloning that doesn't require an internet connection.

I am struggling to implement it into my workflow though, simply because I
do not have time to figure out the simplest things. (full time university
student) Right now I simply do not have the 200 minutes available to figure
this out. Yet I can see that if I took the time to do my own setup it would
save me far more time in the future. Leo would be a wonderful platform for
writing anything, but especially for fiction. If there was a tutorial on
setting up Leo to do the following, I know there would be immediate
interest from people who write on Linux as there simply is not an outliner
that meets the needs of the average writer.

I need basic text formatting at the standard hot key level. ie. Ctrl-B for
bold, Ctrl-I for italic etc. while in the edit window. I know this can be
done, I almost had it working once but then got busy and by the time I came
back it had escaped me how I had done it. I don't need buttons, I don't
need WSIWYG, RST markup works fine as it gives me lots of output options. I
just need the ability to put my head down and write, using the shortcuts
that are baked into my fingertips at this point, and not having to relearn
those keys. Support for standard word-processing keys are important for
writers. :-)

I think that by creating tutorials that address the basics for different
groups of users who are not programmers, Leo could be an extremely popular
program. I would be willing to contribute to such an effort for writers if
I can get a workflow setup that makes sense for me. My current editor of
choice is Scrivener for Windows run using Crossover. No cloning. :-(

Leo is far too complex for the average person who simply wants to
accomplish a task. Simplifying things for specific classes of users would
open Leo up to many more people who share those requirements and may be
interested in learning the rest of the power of Leo as their use of it
increases. Give people a foot in the door and they will figure out how to
open it the rest of the way as needed.
.. @+node:ekr.20131001045038.19039: *5* Chris George
i.e. RST markup for "bold".  Is that the goal?

Cheers -Terry

Almost. That plus the ability to be writing along and hit
Ctrl-b s u d d e n l y Ctrl-b and continue on.

I had Leo working to the extent that I could type suddenly Ctrl-Alt-b and
have it retroactively add the markup but it was simply too much of a leap
to change my work habits. Body memory takes years to develop and is not
trivial to attempt to change.

A set of keybindings that mimic the standard word processing key bindings
that then markup the text using RST or Markdown would be a huge step
forward for me in using Leo. Right now I try to find time to fiddle and
play with it, but that has been going on for almost two years now and I
have yet to commit to any serious work in Leo.

===== Terry

Which boils down to Ctrl-b inserting '*' if there's no selection?

I guess it does.

It may seem silly, as I could just type the *, but Shift-8 isn't baked into
muscle memory, Ctrl-b is.

It boils down to being able to replicate those standard word processing
shortcuts in the edit pane and having them not conflict with other key
combos in Leo. Abstracting it to other markup languages would be cool as
well, some people like RST, some people like Markdown. Everyone can use
Ctrl-b.

Ckeditor looks interesting, but it isn't the buttons that matter. I rarely
use the mouse for marking up text.

==== Terry

Quick stab at
https://github.com/leo-editor/snippets/blob/master/ctrl_b_i_u.py
not very tested.

.. @+node:ekr.20131001045038.19040: *5* Terry's new code
In you myLeoSettings.leo file, under @settings-->@keys-->@shortcuts,
add the shrotcuts (in the body text of the '@shortcuts' node:

markup_inline_bold ! body = Ctrl-B
markup_inline_italic ! body = Ctrl-I
markup_inline_underline ! body = Ctrl-U

restart Leo

Put the downloaded code in a node, and run it... normally you'd do that
by hitting Ctrl-B, but we just redefined that :-)  So use alt-x
execute-script instead.

Now in the body you should be able to get

**bold** *italic* :ul:`underline`

with the expected keys.  It should be fairly predictable when there's a
selection active.  When there isn't, it just opens or closes the markup
type depending on whether the last action was an open or close, so I
guess nesting would be problem, but rst doesn't understand nesting
inline markup anyway.

The difference between open and close only applies to underline, where
the beginning and end delimiters are different - more a demo than
anything.

To make the code active without the execute-script step it could be
placed in a node which starts with @script, although you need
@bool scripting-at-script-nodes = True
under your @settings node then.

Ideally it would be in a plugin (although it seems heavy to put
essentially one function in a plugin) or core, so it didn't need its
own node in the target outline or @settings.
.. @+node:ekr.20131001045038.19041: *5* Chris George
I would like to start putting together everything that it would take to
make Leo the default choice for writers on Linux. If it works for me, it'll
work for many others.
.. @+node:ekr.20131001045038.19042: *5* Terry's new markup_inline plugin
> How difficult would a plugin be?

Easy enough I guess, included in revision 6083, just pushed.

In myLeoSettings.leo, under @settings-->@keys-->@shortcuts::

    markup-inline-bold ! body = Ctrl-B
    markup-inline-italic ! body = Ctrl-I
    markup-inline-underline ! body = Ctrl-U

  - Use Help -> leoSettings.leo to open leoSettings.leo, and copy
    the @enabled-plugins node
    
  - paste this node into your myLeoSettings.leo as a child of the
    @settings node
    
  - just before the line in @enabled-plugins where it says
    "# Alphabetical list of all Leo plugins" add a line:

    markup_inline.py
.. @+node:ekr.20131001045038.19043: *5* Ludwig Schwardt: MacOS brew instructions
Miles's comments inspired me to do something about the Homebrew version of
Leo. Instead of following a long-winded and increasingly out-of-date set of
installation instructions you will soon be able to go (if all goes well!):

brew install leo

I've created a rough formula for Leo and uploaded it to my Homebrew tap. To
try it out, do the following:

- Get Homebrew from http://brew.sh/, following their instructions on how to
install it

- Pay special attention to Python - if you have already installed a bunch
of Python stuff, it's best to keep using the existing Python. Otherwise, if
this is your first Python experience it is safe to install the Homebrew
Python (which will make your life easier) via

  brew install python

  (If all your existing Python tools suddenly disappear, go 'brew remove
  python' to restore order :-))

- Get my formulas:

  brew tap ska-sa/tap

- Et voila:

  brew install leo

- If you kept your old Python, follow the instruction in the caveat printed
at the end of:

  brew info leo

  (I.e. add the suggested line to your ~/.bash_profile file if you are not
  using Homebrew Python)

The formula has three versions:

  brew install leo => installs the latest stable release (4.10)
  brew install --devel leo => installs latest alpha (4.11-a2)
  brew install --HEAD leo => installs the bleeding-edge bzr version from Launchpad

It also installs PyEnchant by default (which can be disabled).

I would love some testing of the installation before I submit it to the
main Homebrew repository (if you guys think the formula is a good idea). I
don't use Leo myself and have only done some cursory poking around.

Miles's point about using a Ruby-based installer to install a Python
package also rings true (although this is becoming more popular these days
on the Mac given how awesome Homebrew is for dependencies and Python
extensions, and this is not any stranger than the C-based "apt-get install
leo").

I would expect to install a Python package such as Leo using "pip install
leo" or "easy_install leo". The main problem has always been that Leo has a
non-standard package layout.

I (only now!) see that Ville, Matt and others have created a setup.py and
leo even exists on PyPI so that these installation commands actually do
something. Unfortunately, for me "pip install leo" results in a Leo that
throws an AssertionError upon running. What is the status of this
installation route? (It is not even mentioned on the installation page!)
Also, "leo 4.10-final" on PyPI actually installs "4.11-devel"...
.. @+node:ekr.20131001045038.19044: *5* EKR
> My (again, personal) feeling is that the current policy right now is: Let
him read the code to understand how it works. So only the users willing to
go through the (1) Learn the physics then (2) develop your own way to
develop "information cities" can actually use leo and use its potential.

​I've spent a lot of time on tutorials, slideshows, etc. However, in light
of your first sentence quoted above, it does seem like I have neglected to
show what people can really do with Leo, at least in the tutorial docs. ​
 
> With Leo, you first have to study and understand what are directives, how
do they work, how everything interacts in the tree, etc. It takes more than
two hours. I would say that it can be measured in weeks or months until you
really know what you are doing.

Maybe, but there is a tantalizing possibility. ​I prototyped Leo in about
two hours, using the MORE outliner as a prototype, and inventing @others in
the process. (I was already deeply involved with sections and section
references.)

So for *me*, the Leo aha was almost instantaneous, once I had an outliner
to prototype my thoughts. Since then, things have become more "voluminous",
yet Leo's DOM is so much simpler and more powerful than it was in the early
days. Do these improvement make Leo *more* difficult to learn? It's hard to
believe so, but then nobody is closer to Leo than I.

> And here is where the "information cities" come again. If Leo were
provided with an interactive Leo file, which would guide you ​...

​The idea of putting a cheat sheet into the default workbook.leo is a great
one. There will be a default_workbook.leo in leo/docs, and the code that
creates ~/workbook.leo will make a copy of the default file. This will
allow continuous improvement on default_workbook.leo.
.. @+node:ekr.20131001045038.19045: *5* EKR
> For marketing, packaging, features, documentation, tutorials,
extensions,... - do a side by side comparison of Vim and Leo - and the
answer(s) to "why leo isn't more popular" leap out at you.


​The answer is not so clear cut.​ Consider the first item returned by
Google for the search "emacs tutorial":
http://www2.lib.uchicago.edu/keith/tcl-course/emacs-tutorial.html

This is hardly a breathtaking introduction to emacs. It would not *by
itself* convert people to emacs. It doesn't even highlight what I consider
the most important Aha about emacs, namely that you don't have to remember
(or type) commands with long names!

No, the reason emacs is popular is that it continues to be taught as *the*
programming editor to generations of computer-programming students. The
students learn *from each other*, using a tedious "tutorial" as a
reference. They learn from looking over each other's shoulders.

Leo would be just as popular, imo, if thousands of college students learned
Leo from each other. This does not excuse lack of further work on Leo
tutorials, but this *is* the essence of the situation.
.. @+node:ekr.20131001100335.15927: *5* Links to Book about Leo
This e-book has a whole sub-heading on using Leo for Joomla:
http://www.gandsnut.net/downloads/Beginning_Joomla!_From_Novice_to_Professional.pdf

amazon link:
http://www.amazon.com/Beginning-Joomla-From-Novice-Professional/dp/1590598482/ref=sr_1_1?ie=UTF8&qid=1380681874&sr=8-1&keywords=beginning+joomla+from+novice+to+professional
.. @+node:ekr.20131001100335.15926: *5* derwish
To me, it's to directed acyclic graphs what MS Excel is to tables. 
.. @+node:ekr.20131012060912.16765: *5* matt
What an enormously long and interesting thread!

Thanks for the sub-thread on using Leo for plain ol' writing. It rings true
for me. Leo doesn't leverage much of my prior muscle memory, and that leads
me away from it, and once away it takes awhile to return. This is still
true even though I've been using Leo for half a decade now, and have even
contributed a small thing or two to the bzr repository.

I generally use notepad2, notepad++ for quick and dirty scripts (usually
Windows batch files), editing .ini,.conf, xml etc., etc. files. Way fast
startup time and their syntax highlighting know more file types and is more
reliable (complete). They also grok (some) regular expressions.

I use vim (+cream) when I want to do search and replace, which happens a
lot. I just never figured out how to use this properly in Leo;
`[esc]:%s/old-thing/new-thing/g` is just so much faster (modulo the
significant time it takes to construct regular expressions that actually
work!). Saved mini-buffer commands across sessions is also very handy. If
vim+cream was faster to start I'd never use the notepad replacements.

Weirdly, even for python I often use PyScripter instead of Leo. Sometimes
it's because I need to write for a different version of python than I'm
running Leo from. Others it's because the output I get from running a Leo
node as a script differs from the same thing from a cmd
shell/IDLE/PyScripter console. I also find PyScripter's code completion a
bit more intuitive for the way I type (though, I didn't discover that until
I'd already wandered away from Leo for a time).

None of this is meant as a slam against or complaint about Leo! It's just
sharing observations of my personal use/not-use patterns.

Part of my difficulty in becoming comfortable with Leo is that I'm also not
that comfortable with python, or even general programming. I'm not a
software developer; this is changing, but very slowly. Consequently I spend
a lot of in-Leo time in a state of confusion, often unsure whether I'm
trying to learn programming or python or Leo or more about the actual
problem I'm started out trying to solve this morning. :)

Anyway, the point touched on earlier about leveraging the skills and
experience people already have before they encounter Leo is pretty key to
the overall adoption rate, I think.

cheers,

-matt
.. @+node:ekr.20101127152442.5878: *3* Url's
@color

@language rest
.. @+node:ekr.20101127154340.5935: *4* Apps
.. @+node:ekr.20101127154340.6806: *5* @url review board: code review
http://www.reviewboard.org/

@language rest
.. @+node:ekr.20101127154340.6824: *5* @url scrivener: cool MacOS outliner
http://www.literatureandlatte.com/scrivener.html

@language rest
.. @+node:ekr.20101127154340.5934: *5* @url tomboy: wiki-like notes
http://projects.gnome.org/tomboy/index.html

http://groups.google.com/group/leo-editor/browse_thread/thread/18d4af19686f2ead
.. @+node:ekr.20101127152442.5879: *4* Autocompletion
.. @+node:ekr.20101127152442.5881: *5* @url ctagscompleter.py (Ville's suggestion)
http://groups.google.com/group/leo-editor/browse_thread/thread/c537f3bc8328a938
.. @+node:ekr.20101127152442.5880: *5* @url pydiction (autocompletion for vim)
http://www.vim.org/scripts/script.php?script_id=850
.. @+node:ekr.20101127154340.5931: *5* @url pysmell: python IDE completion helper
http://code.google.com/p/pysmell/

@language rest
.. @+node:ekr.20101127154340.6828: *5* @url SciTe Java api
http://www.burgaud.com/scite-java-api/

@language rest
.. @+node:ekr.20101127154340.6829: *6* @url tags2api.py: python
http://www.scintilla.org/tags2api.py

@language rest

Produces a .api file for SciTE's identifier completion and calltip features.
.. @+node:ekr.20101127154340.6830: *6* @url generates python api for scite
http://www.koders.com/python/fid7000B9C96CF2C6FB5BCE9DF700365C5B2A1F36A7.aspx?s=gtk#L53

@language rest

gen_python_api.py generates a python.api file for SciTE
.. @+node:ekr.20101127201907.5952: *4* Data bases
.. @+node:ekr.20101127154340.6856: *5* @url lazysoft: cool db (sentences)
http://www.lazysoft.com/

@language rest
.. @+node:ekr.20101127201907.5953: *5* @url persistent trees: couchDB
http://eclipsesource.com/blogs/2009/12/13/persistent-trees-in-git-clojure-and-couchdb-data-structure-convergence/

@language rest


Thread: Interesting post on data tree design

http://groups.google.com/group/leo-editor/browse_thread/thread/e4646371478cd30/44eba97b45bd53b3
.. @+node:ekr.20101127201907.5947: *4* Docs
.. @+node:ekr.20101127201907.5949: *5* @url new design for colorer
http://groups.google.com/group/leo-editor/browse_thread/thread/9fb569af95eee493/00c3bbe120567771

Thread: A new design for the incremental colorer

This long posting will discuss a new design for an incremental
colorizer using QSyntaxHighlighter.  The essential features of this
design became apparent during yesterday's walk. 
.. @+node:ekr.20101127201907.5945: *5* @url docs for my successor
http://groups.google.com/group/leo-editor/browse_thread/thread/440bbf170787c5ed/c228a9fd1d429fa6

@language rest
.. @+node:ekr.20101127201907.5946: *5* @url Leo's MVC architecture
http://groups.google.com/group/leo-editor/browse_thread/thread/6b77a59a3a5c7cbb/f26164f24bee68d2

@language rest
.. @+node:ekr.20101127201907.5948: *5* @url one damn fine checkin
http://groups.google.com/group/leo-editor/browse_thread/thread/278442febf1a1965/6a6640976c23935f

@language rest

configure_tags killing performance again

EKR:

Rev 2119 contains what appears to be a major improvement in speed:

- leoQTextEditWidget.SetAllText now sets a lockout that prevents a duplicate
  recoloring of text. This doubles the speed of the syntax colorer, and almost
  doubles the speed of unit tests!

- updateSyntaxColorer and colorize set self.flag = False for large body text.
  This doesn't prevent recolor from being called, but it does short-circuit
  recolor.

And yes, configure_tags is called only by the ctor, that is, just once. 

Ville:

Now that was one damn fine checkin... 
.. @+node:ekr.20101127201907.5951: *5* @url Stupendous Aha re unit tests
http://groups.google.com/group/leo-editor/browse_thread/thread/47dfb2e1767d2cda/8e658c5b73406a8d

@language rest

I have gotten *zero* responses to my 42 post:
http://groups.google.com/group/leo-and-pylint/browse_thread/thread/3c...

I should have remembered that nobody reads long posts. So here is the
Aha in a nutshell:

Unit tests are not just for testing!  They are *the* master tool for
programming, design, testing, refactoring, studying code, or *anything
else*. 
.. @+node:ekr.20101127154340.6805: *4* Editors
.. @+node:ekr.20101127154340.6807: *5* @url pyxides
http://groups.google.com/group/pyxides

@language rest
.. @+node:ekr.20101127154340.6813: *5* Vim
.. @+node:ekr.20101127154340.6814: *6* @url vim cheat sheet
http://www.fprintf.net/vimCheatSheet.html

@language rest
.. @+node:ekr.20101127154340.6815: *6* @url vim reference guide
http://www.dc.turkuamk.fi/docs/soft/vim/vim.html

@language rest
.. @+node:ekr.20101127154340.6816: *6* @url slight advance intro to vim
http://linuxgazette.net/152/srinivasan.html

@language rest
.. @+node:ekr.20101127154340.6817: *6* @url voom: vim 2-page outliner
http://vim-voom.webs.com/

@language rest
.. @+node:ekr.20101127154340.6818: *6* @url why do those nutheads use vim
http://www.viemu.com/a-why-vi-vim.html

@language rest
.. @+node:ekr.20101127154340.6819: *6* @url viper
http://www.delorie.com/gnu/docs/emacs/viper.html

@language rest
.. @+node:ekr.20101127154340.6820: *6* @ulr learning vim the pragmatic way
http://jrmiii.com/2009/03/06/learning-vim-the-pragmatic-way.html

@language rest
.. @+node:ekr.20101127154340.6822: *5* @url python-UNO (Open Office)
http://udk.openoffice.org/python/python-bridge.html

@language rest

http://wiki.services.openoffice.org/wiki/Documentation/DevGuide/ProUNO/Professional_UNO
.. @+node:ekr.20101127154340.6823: *5* @url open komodo forums
http://community.activestate.com/forums/komodo/open-komodo

@language rest

archives:

http://lists.openkomodo.com/pipermail/openkomodo-dev/
.. @+node:ekr.20101127154340.6825: *5* @url org-babel
http://orgmode.org/worg/org-contrib/babel/intro.php

@language rest
.. @+node:ekr.20101127152442.5884: *4* Feature requests
.. @+node:ekr.20101127152442.5885: *5* Better find/replace dialog
For newbies (and for power users too, for that matter), it may be helpful to
have a "Find and Replace" dialog box similar to the one below:

.. image:: c:/prog/findReplace.jpg

.. @+node:ekr.20101127152442.5886: *5* @image find/replace dialog
c:/prog/findReplace.jpg
.. @+node:ekr.20101127154340.6808: *4* Graphics
.. @+node:ekr.20101127154340.6843: *5* @url inkscape: open source svg editor
http://inkscape.org/

@language rest

.. @+node:ekr.20101127154340.6835: *5* @url interactive map of Linux kernel
http://www.makelinux.net/kernel_map

@language rest
.. @+node:ekr.20101127154340.6836: *5* @url blender: open source content creation
http://www.blender.org/

@language rest
.. @+node:ekr.20101127154340.6809: *5* @url tinkerpop: graphics tools
http://www.tinkerpop.com/
.. @+node:ekr.20101129064803.6061: *5* @url problem solving with graph traversals
http://groups.google.com/group/leo-editor/browse_thread/thread/2e1b240b023b545e/8e9164b52ff25199

Cool slide show: very technical, about graph traversals, from AT&T technical talk

http://www.slideshare.net/slidarko/problemsolving-using-graph-traversals-searching-scoring-ranking-and-recommendation
.. @+node:ekr.20101127152442.5887: *4* Gui
.. @+node:ekr.20101127152442.5888: *5* @url cool hack: detach body editor
http://groups.google.com/group/leo-editor/browse_thread/thread/8616f4e171e1a24b

@language rest

This features a long an important discussion of using Leo with vim.
.. @+node:ekr.20101127152442.5889: *5* @url fast syntax highlighting
http://groups.google.com/group/leo-editor/browse_thread/thread/bdcbe3a4ffb5ac61/7dbeea85f671cfe9

@language rest

Source-highlight Qt Library 0.2.2
http://srchiliteqt.sourceforge.net/source-highlight-qt.html

This seems to be a (c++ based) syntax highlighting package that would
allow us to avoid running any python code for syntax highlighting.
Needless to say this is probably very fast.

Investigation needed how easy this would be to deploy in python & leo.
I might(!) make a plugin that allows you to use it instead of leo's
own highlighters...

t seems to support lots of languages already:

http://www.gnu.org/software/src-highlite/

seems you can define your own languages:

http://www.gnu.org/software/src-highlite/source-highlight.html#Language-Definitions

lots of stuff there, but the simple case (keywords) seem to be simple:

http://www.gnu.org/software/src-highlite/source-highlight.html#Simple-definitions

EKR: We would have to support coloring of Leo directives.

.. @+node:ekr.20101127152442.5890: *5* @url code bubbles: an alternative to clones
http://groups.google.com/group/leo-editor/browse_thread/thread/d030a9eccfae2aa2/3a951894e0c15f72

@language rest

This is a cool video:

http://www.cs.brown.edu/people/acb/codebubbles_site.htm
.. @+node:ekr.20101127154340.6840: *5* @url SIKULI: test gui's
http://groups.csail.mit.edu/uid/sikuli/

@language rest
.. @+node:ekr.20101127152442.5882: *4* Leo projects
.. @+node:ekr.20101127152442.5883: *5* @url @data contextmenu_commands
http://groups.google.com/group/leo-editor/browse_thread/thread/e236feb5bd2a097a/d261f6dbdb7eb950

@language rest

As requested, there is now user-friendly way of adding minibuffer
commands to context menu (in trunk, qt only).

I also added example to quickstart.leo, but all you need to know is
that you need to create

@settings
  @data contextmenu_commands

And the content is like:

# The format is <command> SPACE <description>

stickynote Create a sticky note
read-at-file-nodes Read file nodes 
.. @+node:ekr.20101127152442.5912: *5* @url fuse: Leo as a file system
http://leo.zwiki.org/LeoAsAFileSystem

@language rest

.. @+node:ekr.20101127154340.5936: *5* @url proto of a Leo forum
http://groups.google.com/group/leo-editor/browse_thread/thread/db6e75d82da4b41d

@language rest

A few buttons turns Leo into a competitor for google groups.

.. @+node:ekr.20101129064803.6062: *5* @url stickynotes_plus and markdown
http://groups.google.com/group/leo-editor/browse_thread/thread/f8234a8fdeb08d22/4f42eafa955eae9b

Rich text editing in Leo

markdown: http://www.freewisdom.org/projects/python-markdown/

his is a Python implementation of John Gruber's Markdown. It is almost
completely compliant with the reference implementation.

http://daringfireball.net/projects/markdown/

Markdown is a text-to-HTML conversion tool for web writers. Markdown allows you
to write using an easy-to-read, easy-to-write plain text format, then convert it
to structurally valid XHTML (or HTML).
.. @+node:ekr.20101129064803.6063: *5* @url leo+emacs+pida
http://groups.google.com/group/leo-editor/browse_thread/thread/f65470074f6573ab/c73d5b313526d5af

pida: http://pida.co.uk/

PIDA is an IDE (integrated development environment). PIDA is different from
other IDEs in that it will use the tools you already have available rather than
attempting to reinvent each one. PIDA is written in Python with the PyGTK
toolkit, and although is designed to be used to program in any language, PIDA
has fancy Python IDE.
.. @+node:ekr.20101129064803.6065: *5* @url leoremote.py
http://groups.google.com/group/leo-editor/browse_thread/thread/66bcdc5cac03f5ad/0528c66838f1726d

communicating with a running Leo

@language rest

Terry Brown

For some time I've wanted a simple way to make an running leo session
load a file from the command line, like emacs-client.\

Now, thanks to Ville, I also want a way to make an running leo session
pop up a stickynote window (which would correspond to an automatically
created node which would probably use current data and time as a
headstring).

Ville

First working version of this (using qt) is now on trunk.

Testing it requires some manual work:

1. bzr pull
2. Put this script (it's a simple example client) to your leo-editor
directory: http://pastebin.com/f2bcbfd36
3. Enable leoremote.py plugin
4. Launch leo, do alt-x leoserv-start. The leo process where you last
ran leoserv-start is always the session that will receive your
commands
5. Try running the script (it's a python script, not leo script) you
created at #2 from another command prompt 

Terry

Very very cool Ville, thanks.

Here http://pastebin.com/mf678e24 is my version of a functioning
stickynote script. 

And here: http://pastebin.com/m236194c

is a quick hack at a script to edit (or create and edit) a file in leo
from the command line. 

Terry -> EKR

> Can you explain in more detail.  I am totally lost about why this
> gives us anything new.  Thanks.

It's probably of little benefit to people who work with mouse, menus, and icons
all the time. But if you do everything from the command line (i.e. your OS's
shell), then it makes moving things into leo much smoother.

Suppose I've run leo and checked my todo items for the day, and now
leo's buried under some other window and I'm working in the shell in
some directory I've just created where I've just unzipped something and
now I want to edit a file that was in the .zip.

I can either:

  - find the leo window
  - insert a node
  - active the open file dialog
  - navigate to the directory containing this file
  - select the file
  - and finally do the editing I want to do

or, with Ville's communication to the running leo:

  - enter on the command line `led foo.txt`
  - and do the editing I want to do

where led is a script which causes the running leo to create an @edit
node containing foo.txt and pop to the front with the node selected.

Previously I was much more likely to use emacs, just because it was
easier to invoke that way from the command line.

So, opening files, creating sticky notes, invoking leo to handle output
from grep or diff or whatever - all these things are better now. 


Matt Wilkie

The corresponding point and click process for this scenario is

a) select > r-click > Edit with Leo
b) or drag'n'drop from folder to Leo icon on task bar (or window if visible)

In short, I see this being a productivity boost for all users. 
.. @+node:ekr.20101129064803.6066: *6* @url  Ville's script
http://pastebin.com/f2bcbfd36

from leo.external import lproto
import os

addr = open(os.path.expanduser('~/.leo/leoserv_sockname')).read()
print "will connect to",addr
pc  = lproto.LProtoClient(addr)
pc.send('''

g.es("hello world from remote") 
c = g.app.commanders()[0]

''')
.. @+node:ekr.20101129064803.6067: *6* @url Terry's led script
http://pastebin.com/m236194c
.. @+node:ekr.20101127152442.5903: *5* Ville
.. @+node:ekr.20101127152442.5904: *6* @url use json to represent trees
http://mail.google.com/mail/#label/Cool/121a1f23e75fc348

@language rest

Often, it' useful to decouple node creation from data processing. Case
in point is the read code & the hash based speedup scheme I described
(and referred to again in previous thread!)

(for every instance of 'json' below, you can substitute "hierarchical
python object". I'm thinking of json because of interoperability &
human-readablity of json vs. pickle).

What I mean is:

- Provide a way to create tree structure from json object. This is
easy, just add function

c.createSubTreeFromJson(p, json)

The 'json' arg is just a string with subtree encoded in json. Something like

[ ('h1', 'body1', 'gnx1', [ ('h1.1' , 'body1.1' , 'gnx1.1'), ('h1.2' ,
'body1.2' , 'gnx1.2')], ('h2', ......]

i.e. it's a recursive data structure with list of nodes as tuples (h,
b, gnx, children) where 'children' is the recursive part. So far, so
good. That's the easy part - it's just like xml, apart from the fact
that xml is overkill for this problem, requiring nontrivial parser
code (and thus being of lower usability & speed). Yes, the json is
pretty much the same as repr(python_object).

Now, for the important part.

All code that creates trees (read code as the most relevant example!)
probably shouldn't create the tree directly. Rather, it should use c
and p to learn what it needs to, then compose a result json object and
pass it on to c.createSubTreeFromJson(p, json). This is important &
elegant distinction from current behaviour where the data processing
code also modify the tree. We can basically have tests for most of the
stuff without altering the tree in any way, allowing the tree to stay
"read only" as long as possible.

The hash speedup I've been talking about will execute the auto/thin
parsing code exactly once for the particular file content, store away
the json, and just read that json file on subsequent runs. There is no
need to speculate on feasibility of this scheme, we've already had
that particular discussion. It really is the simplest & fastest
possible solution.

We could also add lazy loading (though this is not as important), but
*also this* will be easier with this scheme. Basically, the lazy
loading would:

- Run in a thread in the background, collecting stuff from external
files to json objects
- Every once in a while, do the "destructive" thing and update the
tree from these json objects, redrawing if necessary.

Also this will be perfectly safe - the background thread never does
gui manipulation. At any given time, it's only reading a file, and
writing the json obj to buffer. No communication with leo gui is
necessary. If position with root obj no longer exist, we just throw
the json away and rescan if needed.
.. @+node:ekr.20101127152442.5905: *6* @url dump snippets as yaml
http://mail.google.com/mail/#label/Cool/121b225322eb0a6a

@language rest

I experimented with creating a "direct" recursive data structure for leo trees.

It's pretty simple (who needs Haskell & friends when we have python):

def p_to_obj(p):
   return [p.h, p.b, p.gnx, [p_to_obj(po) for po in p.children_iter()]]

Yes, I removed utf8 encoding to keep it more elegant ;-).

The whole snippet is somewhat like this:

http://pastebin.com/f6a30f859

It creates a python object like this:

http://pastebin.com/f17a3a285

(list of p,h,gnx, children...)

Now, the interesting part. This can be directly encoded as Yaml, to
render the rather readable:

http://pastebin.com/f3d750af3

The part that needs work is the body text, it should probably be
improvable by using "block scalars".

This idea is related to my "jsonification" refactoring suggestion a
while ago. If functions like read code emitted stuff that p_to_obj
returns, the yaml structure could be used to debug / trace the data
(but still in machine-readable form). This could also be used to
communicate outline structure so that it can be both read, and copy
pasted from emails to real tree structure.

No need to take any action regarding this email. Just a food for
thought, perhaps for inspiration...
.. @+node:ekr.20101127152442.5906: *6* @url creating debian packages
http://groups.google.com/group/leo-editor/browse_thread/thread/f83704ecc4ba225a/9e75ecc105aab953

@language rest

> Would it be possible to publish somewhere the scripts or other files you use
> to create the .deb files?

It's somewhat of a nontrivial effort (that I do manually), but the
debian/ directory is here:

https://code.launchpad.net/~leo-editor-team/leo-editor/packaging-jaunty

In order to build the package, I have set up the ubuntu "ppa" (you can
register it at launchpad), use "debuild -S -sa" to build the package
when everything is at correct place (it's picky about directory & file
names and such), and use "dput" to upload it to launchpad servers.
After a while (one hour?), the package is built.

I have a .leo file that I use to trace these steps. I can push it to
trunk. But there is no automatic script to do it; it's possible to
just run "dpkg-buildpackage -rfakeroot" when the debian/ directory is
at leo-editor directory, but that's not the "proper" way, as your
environment is almost never clean enough to get a reliable package
that directly reflects the source snapshot at that tag. 
.. @+node:ekr.20101127154340.5927: *6* @url objtrees: object trees for unit testing
http://groups.google.com/group/leo-editor/browse_thread/thread/32c3a295d2dae35b
.. @+node:ekr.20101127154340.6852: *6* @url codewise: ville's code completer
http://www.mail-archive.com/leo-editor@googlegroups.com/msg10145.html
.. @+node:ekr.20101129064803.6060: *6* @url server code to interact with a running Leo
http://groups.google.com/group/leo-editor/browse_thread/thread/278aa85d7298a319/ef6446fcf6268c0d

leoRemote.py
.. @+node:ekr.20101127154340.6826: *4* Leo urls
.. @+node:ekr.20101127154340.6827: *5* @url NSIS manual
http://nsis.sourceforge.net/Docs/Contents.html

@language rest
.. @+node:ekr.20101127154340.6833: *5* @url leo-editor-files
http://tinyurl.com/35ddr4w

@language rest
.. @+node:ekr.20101127154340.6851: *5* @url upload to source forge
https://sourceforge.net/project/admin/explorer.php?group_id=3458

@language rest
.. @+node:ekr.20101127154340.6837: *4* Math
.. @+node:ekr.20101127154340.6838: *5* @url sage: mathematics software system
http://sagemath.org/

@language rest
.. @+node:ekr.20101127154340.6839: *5* @url viewdog: viewer for math functions
http://gul.sourceforge.net/viewdog-manual/node3.html

@language rest
.. @+node:ekr.20101127152442.5910: *4* Python tools
.. @+node:ekr.20101127152442.5911: *5* @url clonedigger: find similar code
http://clonedigger.sourceforge.net/
.. @+node:ekr.20101127154340.5928: *5* @url pyexpect: spawn child apps
http://pexpect.sourceforge.net/pexpect.html

@language rest

pexpect: a Python module for spawning child applications and
controlling them automatically. 
.. @+node:ekr.20101129064803.6059: *5* @url pynotify: wait for a file to change
http://groups.google.com/group/leo-editor/browse_thread/thread/2b6cceebd7cd2e3/dddcb73e2a6469b9

@language rest

Linux only

For using inotify from Python, I've used pyinotify; it seems to be a
bit more mature: 

http://pypi.python.org/pypi/pyinotify/0.9.1

There's also inotifyx: (more portable?)

http://pypi.python.org/pypi/inotifyx/0.1.1 
.. @+node:ekr.20101127154340.6804: *5* @url python trace module
http://docs.python.org/library/trace.html

@language rest

See the child node for a cool script.
(It is also in scripts.leo).
.. @+node:ekr.20101127154340.6803: *6* Call hierarchy tracing (using python 'trace' module)
@language python
@tabwidth -4

"""
Analyzing program flow.

Run (ctrl+b) this script after 

cd ~/leo-editor
python -m trace --trackcalls launchLeo.py --gui=qt >trace.txt

"""

tracefile = '~/leo-editor/trace.txt'

import os

tr = open(os.path.expanduser(tracefile))
print tr
top = p.insertAsLastChild().copy()
top.h = 'trace session'
cur = None
no = None
for l in tr:
    if l.startswith('***'):
        cur = top.insertAsLastChild().copy()
        cur.h = os.path.basename(l.split()[1])
    elif l.startswith('  -->'):
        no = cur.insertAsLastChild().copy()
        no.h = os.path.basename(l.split()[1].strip())
    else:
        if no:
            no.b += l.strip() + '\n'

    print ".",
.. @+node:ekr.20101127154340.5932: *5* @url pythoscope: create unit tests automatically
http://pythoscope.org/

@language rest
.. @+node:ekr.20101127154340.5933: *5* @url rope: refactoring library
http://rope.sourceforge.net/

@language rest
.. @+node:ekr.20101127152442.5891: *4* Scientific
.. @+node:ekr.20101127152442.5892: *5* @url calioPY Scientific environment using Leo
http://www.caliopywork.org/
.. @+node:ekr.20101127154340.6812: *5* @url reinteract: scientific platform
http://fishsoup.net/software/reinteract/

@language rest

.. @+node:ekr.20101127152442.5893: *4* Text
.. @+node:ekr.20101127152442.5894: *5* Related to viewrendered plugin
.. @+node:ekr.20101127152442.5895: *6* @url enthought editor for restructured text
http://blog.enthought.com/?p=127

rst project of interest 2009/07/14

http://mail.google.com/mail/#label/Cool/1227cc2c66c976ec

It may be that the codebase could be worth looking at.  I find the
prospect of doing a full sphinx rendering of a document in real time
quite fascinating :-).

Requires ETS (Enthought Tool Suite)
http://code.enthought.com/projects/index.php
.. @+node:ekr.20101127152442.5896: *6* @url autosphinx (related to enthought editor)
https://code.launchpad.net/~villemvainio/leo-editor/autosphinx

http://groups.google.com/group/leo-editor/browse_thread/thread/f292f7c9f2fd66d8

bzr branch lp:~villemvainio/leo-editor/autosphinx 

@language rest


No, we want *another window*, like with @button rst-preview, that is
updated in real time. The html output is not editable anyway.

>> Now, the thing is to
>> update this in real time (as you are editing, i.e. not on save or
>> explicit tangle step).

> How about doing it at idle time, that is, at most every half second?

It needs to happen in another process in order to not bog down normal
editing (if we stop doing that, the ui will hang). Generating sphinx
output is somewhat expensive operation.

> I don't want to involve any part of the file logic in rendering: it's too
> complex as it is.

With @auto-rst, we get this for free (i.e. rendering every time you
save). Barring that, we could do the rendering just for current node,
but that is not faithful to the final output.

I think the best implementation is a new process that monitors a file.
That way, it's not really leo-specific, and it's easy to do. Leo just
needs to tell it what file to monitor.

-- 
.. @+node:ekr.20101127152442.5913: *6* @url manuel: rST testing tool
http://pypi.python.org/pypi/manuel

@language rest

Manuel parses documents, evaluates their contents,
then formats the result of the evaluation.
.. @+node:ekr.20101127152442.5897: *5* @url controlling tex parameter from rst markup
http://groups.google.com/group/leo-editor/browse_thread/thread/20ec9f3ec33eb174/dacfce8823e727c1

@language rest

.. @+node:ekr.20101127152442.5898: *6* solution 1: use @raw
@language rest

> On a related note, is there a good example somewhere of embedding
> LaTeX markup in rst?

You can use the ..raw directive

Here is a quote from another thread
http://groups.google.com/group/leo-editor/browse_thread/thread/16272e...

QQQ
One of the difficulties was how to "convince" Leo to write raw code, I
wanted to write some equations and tables, so it was necessary to use
symbols like \, { }, and so on, i. e., what is required by LaTeX. The
problem was that all these symbols were "interpreted" wrongly and
replaced by an unreadable code. The solution was to use the raw
directive. An example is much better to show this. Assume the
following LaTeX code for an equation and a table in a Leo node:

@
.. raw:: latex html

 \begin{equation}  f(x)=\frac{e^{X\beta}}{1-e^{X\beta}}  \end
{equation}

 \newline
 \begin{tabular}[b]{|r|l||c|r|}
    <content deleted ...>
 \end{tabular}

@c

Attention: RsT code requires an indentation with respect to the raw
directive. I didn't know it. It is here a leading blank space for all
LaTeX code inside the raw directive. And then it worked fine.
QQQ 
.. @+node:ekr.20101127152442.5899: *6* solution 2: use make
I think you get the most control by having leo generate an rst file and
then processing it yourself.

Like this somewhat dated page (skip the first section):
http://leo.zwiki.org/RstEmacs

For a project I'm currently working on, currently :), my set up is like
this:

Edit rst in leo using an @auto-rst node.

Run this script (it is (pointlessly) a Makefile):

.. sourcecode:: make

  all:
        rst2html.py report2.rst report2.html
        itex2MML < report2.html > report2.xhtml
        rst2latex.py --documentoptions=letterpaper --stylesheet-path=myprefs.inc \
          --reference-label="ref*" --use-latex-citations report2.rst report2.tex
        echo | pdflatex -draftmode report2 >/dev/null 2>&1
        echo | pdflatex -draftmode report2 >/dev/null 2>&1
        echo | pdflatex report2 2>&1 | tr \\n \\r
        rst2latex.py --documentoptions=letterpaper --stylesheet-path=myprefs.inc \
          report2.rst report2.tex

The two -draftmode parses take care of references etc. much quick than
non-draft, because they don't chew up time turning pngs and pdfs into
part of the pdf output.  I.e. they don't produce pdf output.

``myprefs.inc`` looks like:

.. sourcecode:: latex

  \usepackage{pslatex}
  \usepackage{mathptmx}
  \usepackage[scaled=.90]{helvet}
  \usepackage{courier}
  \renewcommand\sfdefault{phv}%               use helvetica for sans serif
  \renewcommand\familydefault{\sfdefault}%    use sans serif by default
  \renewcommand{\topfraction}{0.85}
  \renewcommand{\textfraction}{0.1}
  \renewcommand{\floatpagefraction}{0.75}
  \setcounter{bottomnumber}{2}
  \renewcommand{\bottomfraction}{0.8}

  \setlength{\parskip}{2ex}
  \setlength{\parindent}{0pt}

  \addtolength{\oddsidemargin}{-1.5cm}
  \addtolength{\evensidemargin}{-1.5cm}
  \addtolength{\textwidth}{3cm}
  \addtolength{\topmargin}{-2cm}
  \addtolength{\textheight}{4cm}

  \usepackage{fancyhdr}
  \pagestyle{fancy}
  \lhead{Synoptic mapping of Native Plant Communities}
  \rhead{}
  \lfoot{\footnotesize 20091203-draft }

itex2MML (referenced in the ``Makefile``) converts latex math notation to MathML in the .xhtml output.

To insert raw latex in the latex output from rst, use

.. sourcecode:: rst

  .. raw:: latex

     \some{latex here}

Cheers -Terry 
.. @+node:ekr.20101127152442.5900: *5* @url rst to anything
http://rst2a.com/

@language rest

A web service that converts reStructuredText to pdf or other formats.

You can use that to, say, quickly generate pdf from an (@auto-) rst file.

There is an api for the web service:

http://rst2a.com/api/
.. @+node:ekr.20101127154340.6845: *5* @url mxTextTools: python text tools
http://www.egenix.com/products/python/mxBase/mxTextTools/

@language rest
.. @+node:ekr.20101127154340.6847: *5* @url 2die4 games: TL (very cool rst stuff)
http://www.2die4games.com/

Thread: TL's 2die4games web site

http://groups.google.com/group/leo-editor/browse_thread/thread/cace15fe101e4844/6acd66a982bc063b
.. @+node:ekr.20101127201907.5950: *5* @url idea - presentation tool
http://groups.google.com/group/leo-editor/browse_thread/thread/4ea2d3f7d2c68106/478c773f875815db

@language rest

- Create one QWebView window, zoom it in to have large fonts
- Create @button that converts current node containing
  restructuredtext to html, and pushes that html to QWebView.

Voila', instant presentation tool. The webview window would be on projector, and
leo would be in your private computer. You can easily edit the text, or find new
interesting slides to present in privacy of your own screen.

=====

Terry:
.. @+node:ekr.20101127201907.5954: *5* @url rst/html in email
http://groups.google.com/group/leo-editor/browse_thread/thread/d119424cbccc96df/379a0600dfb8f4ca?lnk=gst&q=terry+brown#379a0600dfb8f4ca

@language rest

Was: "Anchors" as pseudo-persistent positions

Now: OT: rst/html in email

Important : You should try and view the HTML version of this message!

On Thu, 10 Dec 2009 10:46:05 -0600 Terry Brown <terry_n_brown@yahoo.com> wrote:

> Hey - would it be cool to have an email system which lets you write
> in rst and then sends both text (rst) and html forms...

Despite the fact y'all failed to chorus "Yes it would" ;-) I went ahead and set
it up in Claws-mail

   1.

      Add an rst-preview button to the Claws-mail compose window. It just runs a
      user command on the body text:

      | rst2pyg >~/.tmp.html; x-www-browser ~/.tmp.html

      rst2pyg is included below.
   2.

      Add an indent button to the Claws-mail compose window. It just runs a user command on the selected text:

          | sed 's/^/  /' |

   3.

      Write rst2email, included below.
   4.

      Use Send Later for messages you want to process, so rst2email can get at them in the queue directory.
   5.

      You can get rst2email_pygments.py (as imported by both rst2email and rst2pyg) from my blog, it's the file at the end of this page.

rst2email::

    #!/usr/bin/python
    """rst2email - look for messages in an email queue folder and
    add an html part by processing the text part as rst

    Terry Brown, terry_n_brown@yahoo.com
    """

    import email
    import mailbox
    from email.mime.text import MIMEText
    from docutils.core import publish_string

    # this import adds the sourcecode:: directive to rst
    import rst2email_pygments

    queue = "/home/tbrown/Mail/queue"

    mbox = mailbox.MH(queue, email.message_from_file)

    for msgkey in mbox.iterkeys():
        # d = email.utils.parsedate(msg.get('Date'))

        msg = mbox[msgkey]

        if not msg.is_multipart():
            txt = msg.get_payload()
            html = publish_string(txt, writer_name='html')
            part1 = MIMEText(txt, 'plain')
            part2 = MIMEText(html, 'html')
            part1["X-rst2email"] = "rst"
            part2["X-rst2email"] = "html"
            msg.set_type("multipart/alternative")
            msg.set_payload([])
            msg.attach(part1)
            msg.attach(part2)
            mbox[msgkey] = msg
        else:
            txtpart = None
            htmlpart = None
            for part in msg.walk():
                if part.get_content_type() == "text/plain":
                    if txtpart:  # can't handle more than one
                        txtpart = None
                        break
                    txtpart = part
                if part.get_content_type() == "text/html":
                    if htmlpart:  # can't handle more than one
                        htmlpart = None
                        break
                    htmlpart = part
            if txtpart and htmlpart and not txtpart.is_multipart():
                htmlpart.set_payload(
                    publish_string(txtpart.get_payload(),
    writer_name='html')) txtpart["X-rst2email"] = "rst"
                htmlpart["X-rst2email"] = "html"
                msg.set_type("multipart/alternative")
                mbox[msgkey] = msg

rst2pyg::



    #!/usr/bin/python

    from docutils.core import publish_string
    import rst2email_pygments
    import sys

    print publish_string(sys.stdin.read(), writer_name='html')

===== Ville M. Vainio	

This kind of stuff makes me think html email is actually a somewhat tolerable concept.

For thunderbird, there is pasteCode:

https://addons.mozilla.org/en-US/thunderbird/addon/4046

===== Terry Brown

Maybe :-) even as I wrote it it was more because I thought email
rendered (to html) from rst was cool, rather than that I think html
email is necessary.  It's funny looking at postings in this list vs. my
main inbox, here very very few msgs have html parts, whereas in the
main inbox at least 50% do.

Did occur to me that it would probably be possibly to set up some
combination of unicode chrs and css which would render leo trees
nicely in an html email.
.. @+node:ekr.20101129064803.6064: *5* @url rst/latex tricks
http://groups.google.com/group/leo-editor/browse_thread/thread/d052979864a278bc/3ff8f82b2a80774d

@language rest

python in leo to generate rst

Here's a fun example of rst / latex tricks.  I want (well, the report
recipients want) 16 similar figures included.  I write a little python
in an rst comment, use ``Ctrl-b`` to execute it, and then an rst ``..
include::`` directive to include its output.  Here's the rst:

.. sourcecode:: rst

  The range of each input variable which occurs within each cluster
  could be examined to assign a biological meaning to each cluster.
  Figures `Upland cluster 1 and variables`_ through
  `Wetland cluster 8 and variables`_ show the
  relationships between clusters and variables.  The maps
  give a more general view of the clusters, and
  a variable by variable interpretation of each cluster may
  not add much useful information.

  ..  python

    out = file('/home/tbrown/Desktop/Proj/AitkinMap/varvsclust.rst', 'w')
    def pnt(x): out.write(x+'\n')
    for uw in 'Upland', 'Wetland':
      for i in range(1,9):
        pnt(".. figure:: plots/%s_Cluster_%d.pdf" % (uw,i))
        pnt("   :width: 95%")
        pnt("")
        pnt("   %s cluster %d and variables" % (uw,i))
        pnt("")
    out.close()

  .. include:: varvsclust.rst

Note: The .. python is just a comment.  "Empty" comments like::

    ..
      print 40+2
      print 'done'

don't work, because empty comments don't consume indented blocks
.. @+node:ekr.20101127152442.5901: *4* Video tools
.. @+node:ekr.20101127154340.6841: *5* @url screenr: instant screencasts for twitter
http://screenr.com/
.. @+node:ekr.20101127154340.6842: *5* @url jing
http://jingproject.com
.. @+node:ekr.20101127152442.5907: *4* Web technologies
.. @+node:ekr.20101127154340.6811: *5* @url 0mq: socket library
http://www.zeromq.org/
.. @+node:ekr.20101127154340.6810: *5* @url flask: micro devel framework for python
http://flask.pocoo.org/docs/

@language rest
.. @+node:ekr.20101127154340.6848: *5* @url google app engine
http://code.google.com/appengine/

@language rest
.. @+node:ekr.20101127154340.6849: *5* @url browser shots: web page testing
http://browsershots.org/

@language rest
.. @+node:ekr.20101127154340.6850: *5* @url goosh: google shell
http://goosh.org/

@language rest
.. @+node:ekr.20101127152442.5908: *5* @url json
http://www.json.org/

@language rest

**object**:

{   string : value,
    string : value,
    ...
} 

**array**:

[ value, value...]

**value**: string object array true false null

**number**: 
    int
    int frac
    int e digits
    int frac e digits 
.. @+node:ekr.20101127152442.5909: *5* @url places to ask questions
serverfault.com
stackoverflow.com

@language rest

Quora
.. @+node:ekr.20101127154340.5930: *5* @url pyjamas: AJAX tool kit
http://pyjs.org/

@language rest

Pyjamas: AJAX tool kit.
.. @+node:ekr.20101127154340.6846: *5* @url w3c: web standards page
http://www.w3.org/

@language rest
.. @+node:ekr.20101127154340.6853: *4* Windows
.. @+node:ekr.20101127154340.6854: *5* @url cmd
http://commandwindows.com/command1.htm

@language rest
.. @+node:ekr.20101127154340.6844: *5* @url process monitor for windows
http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx?PHPSESSID=d926

@language rest
.. @+node:ekr.20101127154340.6855: *5* @url windows shortcuts
http://www.codeproject.com/Articles/36538/Windows-7-Tricks-and-Keyboard-Shortcuts.aspx

@language rest
.. @+node:ekr.20161022035203.1: ** Test code: do not delete
@language python
.. @+node:ekr.20160923163813.1: *3* @buttons for pyflakes & flake8
.. @+node:ekr.20160923163813.2: *4* @@button test-pyflakes @key=Ctrl-6
# g.cls()
import os
import pyflakes
import sys
import time
if 1:
    p = c.p
else:
    h = '@file leoApp.py'
    p = g.findNodeAnywhere(c, h)
assert p
@others
if c.isChanged():
    c.save()
PyflakesCommand(c).run(p=p)

.. @+node:ekr.20160923163813.14: *5* class PyflakesCommand
class PyflakesCommand(object):
    '''A class to run pyflakes on all Python @<file> nodes in c.p's tree.'''

    def __init__(self, c):
        '''ctor for PyflakesCommand class.'''
        self.c = c
        self.seen = [] # List of checked paths.

    @others
.. @+node:ekr.20160923163813.15: *6* pyflakes.check_all
def check_all(self, paths):
    '''Run pyflakes on fn.'''
    from pyflakes import api, reporter
    total_errors = 0
    for fn in sorted(paths):
        # Report the file name.
        sfn = g.shortFileName(fn)
        s = g.readFileIntoEncodedString(fn, silent=False)
        if not s.strip():
            return
        r = reporter.Reporter(
            errorStream=sys.stderr,
            warningStream=sys.stderr,
            )
        errors = api.check(s, sfn, r)
        total_errors += errors
        if False and errors:
            # Annoying.
            print('%s error%s in %s' % (errors, g.plural(errors), fn))
    ok = not total_errors
    return ok
.. @+node:ekr.20160923163813.16: *6* pyflakes.find
def find(self, p):
    '''Return True and add p's path to self.seen if p is a Python @<file> node.'''
    found = False
    if p.isAnyAtFileNode():
        aList = g.get_directives_dict_list(p)
        path = c.scanAtPathDirectives(aList)
        fn = p.anyAtFileNodeName()
        if fn.endswith('.py'):
            fn = g.os_path_finalize_join(path, fn)
            if fn not in self.seen:
                self.seen.append(fn)
                found = True
    return found
.. @+node:ekr.20160923163813.17: *6* pyflakes.run
def run(self, p=None):
    '''Run Pyflakes on all Python @<file> nodes in c.p's tree.'''
    c = self.c
    root = p or c.p
    # Make sure Leo is on sys.path.
    leo_path = g.os_path_finalize_join(g.app.loadDir, '..')
    if leo_path not in sys.path:
        sys.path.append(leo_path)
    # Run pyflakes on all Python @<file> nodes in root's tree.
    t1 = time.time()
    found = False
    for p in root.self_and_subtree():
        found |= self.find(p)
    # Look up the tree if no @<file> nodes were found.
    if not found:
        for p in root.parents():
            if self.find(p):
                found = True
                break
    # If still not found, expand the search if root is a clone.
    if not found:
        isCloned = any([p.isCloned() for p in root.self_and_parents()])
        if isCloned:
            for p in c.all_positions():
                if p.isAnyAtFileNode():
                    isAncestor = any([z.v == root.v for z in p.self_and_subtree()])
                    if isAncestor and self.find(p):
                        break
    paths = list(set(self.seen))
    if paths:
        ok = self.check_all(paths)
        g.es_print('pyflakes: %s file%s in %s' % (
            len(paths), g.plural(paths), g.timeSince(t1)))
    return ok
.. @+node:ekr.20160923163813.7: *4* @@button test-flake8 @key=Ctrl-7
# g.cls()
import flake8
import os
import sys
import time
if 1:
    p = c.p
else:
    h = '@file leoApp.py'
    p = g.findNodeAnywhere(c, h)
assert p
@others
if c.isChanged():
    c.save()
Flake8Command(c).run(p=p)

.. @+node:ekr.20160923163813.19: *5* class Flake8Command
class Flake8Command(object):
    '''A class to run flake8 on all Python @<file> nodes in c.p's tree.'''

    def __init__(self, c, quiet=False):
        '''ctor for Flake8Command class.'''
        self.c = c
        self.quiet = quiet
        self.seen = [] # List of checked paths.

    @others
.. @+node:ekr.20160923163813.20: *6* flake8.check_all
def check_all(self, paths):
    '''Run flake8 on all paths.'''
    from flake8 import engine, main
    config_file = self.get_flake8_config()
    if config_file:
        style = engine.get_style_guide(
            parse_argv=False,
            config_file=config_file,
        )
        report = style.check_files(paths=paths)
        # Set statistics here, instead of from the command line.
        options = style.options
        options.statistics = True
        options.total_errors = True
        # options.benchmark = True
        main.print_report(report, style)
.. @+node:ekr.20160923163813.21: *6* flake8.find
def find(self, p):
    '''Return True and add p's path to self.seen if p is a Python @<file> node.'''
    found = False
    if p.isAnyAtFileNode():
        aList = g.get_directives_dict_list(p)
        path = c.scanAtPathDirectives(aList)
        fn = p.anyAtFileNodeName()
        if fn.endswith('.py'):
            fn = g.os_path_finalize_join(path, fn)
            if fn not in self.seen:
                self.seen.append(fn)
                found = True
    return found
.. @+node:ekr.20160923163813.22: *6* flake8.get_flake8_config
def get_flake8_config(self):
    '''Return the path to the pylint configuration file.'''
    trace = False and not g.unitTesting
    join = g.os_path_finalize_join
    dir_table = (
        g.app.homeDir,
        join(g.app.homeDir, '.leo'),
        join(g.app.loadDir, '..', '..', 'leo', 'test'),
    )
    if g.isPython3:
        base_table = ('flake8', 'flake8.txt')
    else:
        base_table = ('flake8',)
    for base in base_table:
        for path in dir_table:
            fn = g.os_path_abspath(join(path, base))
            if g.os_path_exists(fn):
                if trace: g.trace('found:', fn)
                return fn
    if not g.unitTesting:
        g.es_print('no flake8 configuration file found in\n%s' % (
            '\n'.join(dir_table)))
    return None
.. @+node:ekr.20160923163813.23: *6* flake8.run
def run(self, p=None):
    '''Run flake8 on all Python @<file> nodes in c.p's tree.'''
    c = self.c
    root = p or c.p
    # Make sure Leo is on sys.path.
    leo_path = g.os_path_finalize_join(g.app.loadDir, '..')
    if leo_path not in sys.path:
        sys.path.append(leo_path)
    # Run flake8 on all Python @<file> nodes in root's tree.
    t1 = time.time()
    found = False
    for p in root.self_and_subtree():
        found |= self.find(p)
    # Look up the tree if no @<file> nodes were found.
    if not found:
        for p in root.parents():
            if self.find(p):
                found = True
                break
    # If still not found, expand the search if root is a clone.
    if not found:
        isCloned = any([p.isCloned() for p in root.self_and_parents()])
        # g.trace(isCloned,root.h)
        if isCloned:
            for p in c.all_positions():
                if p.isAnyAtFileNode():
                    isAncestor = any([z.v == root.v for z in p.self_and_subtree()])
                    # g.trace(isAncestor,p.h)
                    if isAncestor and self.find(p):
                        break
    paths = list(set(self.seen))
    if paths:
        self.check_all(paths)
    g.es_print('flake8: %s file%s in %s' % (
        len(paths), g.plural(paths), g.timeSince(t1)))
.. @+node:ekr.20160923163813.13: *4* pyflakes command
@g.command('pyflakes')
def pyflakes_command(event):
    '''
    Run pyflakes on all nodes of the selected tree,
    or the first @<file> node in an ancestor.
    '''
    @others
        # define class PyFlakesCommand.
    c = event.get('c')
    if c:
        if c.isChanged():
            c.save()
        if pyflakes:
            PyflakesCommand(c).run()
        else:
            g.es_print('can not import pyflakes')
.. @+node:ekr.20160923163813.14: *5* class PyflakesCommand
class PyflakesCommand(object):
    '''A class to run pyflakes on all Python @<file> nodes in c.p's tree.'''

    def __init__(self, c):
        '''ctor for PyflakesCommand class.'''
        self.c = c
        self.seen = [] # List of checked paths.

    @others
.. @+node:ekr.20160923163813.15: *6* pyflakes.check_all
def check_all(self, paths):
    '''Run pyflakes on fn.'''
    from pyflakes import api, reporter
    total_errors = 0
    for fn in sorted(paths):
        # Report the file name.
        sfn = g.shortFileName(fn)
        s = g.readFileIntoEncodedString(fn, silent=False)
        if not s.strip():
            return
        r = reporter.Reporter(
            errorStream=sys.stderr,
            warningStream=sys.stderr,
            )
        errors = api.check(s, sfn, r)
        total_errors += errors
        if False and errors:
            # Annoying.
            print('%s error%s in %s' % (errors, g.plural(errors), fn))
    ok = not total_errors
    return ok
.. @+node:ekr.20160923163813.16: *6* pyflakes.find
def find(self, p):
    '''Return True and add p's path to self.seen if p is a Python @<file> node.'''
    found = False
    if p.isAnyAtFileNode():
        aList = g.get_directives_dict_list(p)
        path = c.scanAtPathDirectives(aList)
        fn = p.anyAtFileNodeName()
        if fn.endswith('.py'):
            fn = g.os_path_finalize_join(path, fn)
            if fn not in self.seen:
                self.seen.append(fn)
                found = True
    return found
.. @+node:ekr.20160923163813.17: *6* pyflakes.run
def run(self, p=None):
    '''Run Pyflakes on all Python @<file> nodes in c.p's tree.'''
    c = self.c
    root = p or c.p
    # Make sure Leo is on sys.path.
    leo_path = g.os_path_finalize_join(g.app.loadDir, '..')
    if leo_path not in sys.path:
        sys.path.append(leo_path)
    # Run pyflakes on all Python @<file> nodes in root's tree.
    t1 = time.time()
    found = False
    for p in root.self_and_subtree():
        found |= self.find(p)
    # Look up the tree if no @<file> nodes were found.
    if not found:
        for p in root.parents():
            if self.find(p):
                found = True
                break
    # If still not found, expand the search if root is a clone.
    if not found:
        isCloned = any([p.isCloned() for p in root.self_and_parents()])
        if isCloned:
            for p in c.all_positions():
                if p.isAnyAtFileNode():
                    isAncestor = any([z.v == root.v for z in p.self_and_subtree()])
                    if isAncestor and self.find(p):
                        break
    paths = list(set(self.seen))
    if paths:
        ok = self.check_all(paths)
        g.es_print('pyflakes: %s file%s in %s' % (
            len(paths), g.plural(paths), g.timeSince(t1)))
    return ok
.. @+node:ekr.20160923163813.18: *4* flake8 command
@g.command('flake8')
def flake8_command(event):
    '''
    Run flake8 on all nodes of the selected tree,
    or the first @<file> node in an ancestor.
    '''
    @others
        # define class Flake8Command.
    c = event.get('c')
    if c:
        if c.isChanged():
            c.save()
        if flake8:
            Flake8Command(c).run()
        else:
            g.es_print('can not import flake8')
.. @+node:ekr.20160923163813.19: *5* class Flake8Command
class Flake8Command(object):
    '''A class to run flake8 on all Python @<file> nodes in c.p's tree.'''

    def __init__(self, c, quiet=False):
        '''ctor for Flake8Command class.'''
        self.c = c
        self.quiet = quiet
        self.seen = [] # List of checked paths.

    @others
.. @+node:ekr.20160923163813.20: *6* flake8.check_all
def check_all(self, paths):
    '''Run flake8 on all paths.'''
    from flake8 import engine, main
    config_file = self.get_flake8_config()
    if config_file:
        style = engine.get_style_guide(
            parse_argv=False,
            config_file=config_file,
        )
        report = style.check_files(paths=paths)
        # Set statistics here, instead of from the command line.
        options = style.options
        options.statistics = True
        options.total_errors = True
        # options.benchmark = True
        main.print_report(report, style)
.. @+node:ekr.20160923163813.21: *6* flake8.find
def find(self, p):
    '''Return True and add p's path to self.seen if p is a Python @<file> node.'''
    found = False
    if p.isAnyAtFileNode():
        aList = g.get_directives_dict_list(p)
        path = c.scanAtPathDirectives(aList)
        fn = p.anyAtFileNodeName()
        if fn.endswith('.py'):
            fn = g.os_path_finalize_join(path, fn)
            if fn not in self.seen:
                self.seen.append(fn)
                found = True
    return found
.. @+node:ekr.20160923163813.22: *6* flake8.get_flake8_config
def get_flake8_config(self):
    '''Return the path to the pylint configuration file.'''
    trace = False and not g.unitTesting
    join = g.os_path_finalize_join
    dir_table = (
        g.app.homeDir,
        join(g.app.homeDir, '.leo'),
        join(g.app.loadDir, '..', '..', 'leo', 'test'),
    )
    if g.isPython3:
        base_table = ('flake8', 'flake8.txt')
    else:
        base_table = ('flake8',)
    for base in base_table:
        for path in dir_table:
            fn = g.os_path_abspath(join(path, base))
            if g.os_path_exists(fn):
                if trace: g.trace('found:', fn)
                return fn
    if not g.unitTesting:
        g.es_print('no flake8 configuration file found in\n%s' % (
            '\n'.join(dir_table)))
    return None
.. @+node:ekr.20160923163813.23: *6* flake8.run
def run(self, p=None):
    '''Run flake8 on all Python @<file> nodes in c.p's tree.'''
    c = self.c
    root = p or c.p
    # Make sure Leo is on sys.path.
    leo_path = g.os_path_finalize_join(g.app.loadDir, '..')
    if leo_path not in sys.path:
        sys.path.append(leo_path)
    # Run flake8 on all Python @<file> nodes in root's tree.
    t1 = time.time()
    found = False
    for p in root.self_and_subtree():
        found |= self.find(p)
    # Look up the tree if no @<file> nodes were found.
    if not found:
        for p in root.parents():
            if self.find(p):
                found = True
                break
    # If still not found, expand the search if root is a clone.
    if not found:
        isCloned = any([p.isCloned() for p in root.self_and_parents()])
        # g.trace(isCloned,root.h)
        if isCloned:
            for p in c.all_positions():
                if p.isAnyAtFileNode():
                    isAncestor = any([z.v == root.v for z in p.self_and_subtree()])
                    # g.trace(isAncestor,p.h)
                    if isAncestor and self.find(p):
                        break
    paths = list(set(self.seen))
    if paths:
        self.check_all(paths)
    g.es_print('flake8: %s file%s in %s' % (
        len(paths), g.plural(paths), g.timeSince(t1)))
.. @+node:ekr.20161006162035.1: *3* cff regex pattern to find section references
# This works
<<(\s*)(\w+)(\s+)(\w+)(.*)>>

# These don't work
<<(\s*)(?!(import|docstring|includes))(\w+)(\s*)>>
<< xyz >>
<< import >>
.. @+node:ekr.20160504053037.1: *3* test class-based decorators
# test __call__ method

class Test(object):
    def __init__(self):
        g.trace('-----')
    def __call__(self):
        g.trace('=====')

Test()()
.. @+node:ekr.20161017043402.1: *3* test create_json
g.cls()
import imp
import leo.core.leoVersion as v
imp.reload(v)
v.create_commit_timestamp_json()
commit, date = v.get_version_from_git(short=False)
print(commit + ' ' + date)
commit, date = v.get_version_from_json(short=False)
print(commit + ' ' + date)
.. @+node:ekr.20150704140210.1: *3* test leoCheck.py
g.cls()
import imp
import leo.core.leoCheck as leoCheck
imp.reload(leoCheck)
files = (
    [
        r'c:\leo.repo\leo-editor\leo\core\leoNodes.py',
        # r'c:\leo.repo\leo-editor\leo\core\leoApp.py',
        # r'c:\leo.repo\leo-editor\leo\core\leoFileCommands.py',
    ] or
    leoCheck.ProjectUtils().project_files('leo')
)
leoCheck.test(files)
.. @+node:ekr.20150524200357.1: *3* testing beautifier code
.. @+node:ekr.20150521113831.1: *4* @@button TEST PTB
# @language python
r'''The script to test AddTokensToTree class.'''
g.cls()
if c.isChanged():
    c.save()
project = None # 'leo'
<< imports >>
if project:
    # 17.1 sec. --> 17.4 --> 17.3
    # The format time is about twice the tokenize time.
    aList = leoBeautify.ProjectUtils().project_files(project)
    aList = ['@file %s' % (g.shortFileName(z).rstrip()) for z in aList]
    settings_d = {
        # 'output_string': True,
        # 'stats': True,
    }
else:
    aList = [
        'unit test: leoBeautify.py',
        # '@file leoColorizer.py',
        # '@file leoBeautify.py',
        # '@file leoCommands.py',
    ]
    small = aList[0].startswith('unit test:')
    settings_d = {
        'tidy-keep-blank-lines': True,
        'input_string': small,
        # 'input_lines':  True,
        # 'input_tokens': small,
        # 'output_tokens': small,
        'output_string': small,
        'stats': not small # and len(aList) == 1,
    }
assert aList,'no input in %s' % (p.h)
<< init total stats >>
t1 = time.clock()
for h in aList:
    p2 = g.findNodeAnywhere(c,h)
    if p2:
        try:
            b = leoBeautify.test_beautifier(c,h,p2,settings_d)
            << update total stats >>
        except Exception:
            g.es_exception()
if len(aList) > 1:
    << print total stats >>
tot_time = time.clock()-t1
if len(aList) > 1:
    print('done: %4.2f sec. = %4.2f sec./file' % (tot_time,tot_time/float(tot_files)))
else:
    print('done: %4.2f sec.' % (tot_time))
.. @+node:ekr.20150525073117.1: *5* << imports >>
import leo.core.leoBeautify as leoBeautify
import imp
import time
imp.reload(leoBeautify)
.. @+node:ekr.20150530073856.1: *5* << init total stats >>
tot_beautify_time = 0.0
tot_changed_nodes = 0
tot_check_time = 0.0
tot_files = 0
tot_input_tokens = 0
tot_output_tokens = 0
tot_parse_time = 0.0
tot_strings = 0
tot_tokenize_time = 0.0
tot_total_time = 0.0
.. @+node:ekr.20150530073906.1: *5* << update total stats >>
tot_beautify_time += b.beautify_time
tot_changed_nodes += b.n_changed_nodes
tot_check_time += b.check_time
tot_files += 1
tot_input_tokens += b.n_input_tokens
tot_output_tokens += b.n_output_tokens
tot_parse_time += b.parse_time
tot_strings += b.n_strings
tot_tokenize_time += b.tokenize_time
tot_total_time += b.total_time
.. @+node:ekr.20150530074429.1: *5* << print total stats >>
print('========== totals ==========')
print('files          %s' % tot_files)
print('changed nodes  %s' % tot_changed_nodes)
print('tokens         %s' % tot_input_tokens)
print('len(code_list) %s' % tot_output_tokens)
print('len(s)         %s' % tot_strings)
print('parse          %4.2f sec.' % tot_parse_time)
print('tokenize       %4.2f sec.' % tot_tokenize_time)
print('format         %4.2f sec.' % tot_beautify_time)
print('check          %4.2f sec.' % tot_check_time)
print('total          %4.2f sec.' % tot_total_time)
.. @+node:ekr.20150524071341.1: *4* unit test: leoBeautify.py
# Extraneous backslash causes problems.
function1("NOTE: embedded layout in @settings/@data free-layout-layout" \
          "overrides saved layout " + name)
function2(a,
          b)
function3(a, \
          b)
# Weird case: comments ends in backslash\\
# Next comment

def test():
    legend = '''\
legend:
'''
    c = self.c
.. @+node:ekr.20150601042305.1: *5* new op problems
def spam():
    pass

a = b[25]
return {}
.. @+node:ekr.20150601015905.1: *5* return test
def return_test():
    if 1: return
    while True:
        if 2: return
    if 3:
        return
.. @+node:ekr.20150531104223.1: *5* class Command
class Command:
    '''
    A global decorator for functions outside of any class.
    
    g can *not* be used anywhere in this class!
    '''

    def __init__(self, name, **kwargs):
        '''Ctor for command decorator class.'''
        self.name = name

    def __call__(self, func):
        '''Register command for all future commanders.'''
        global_commands_dict[self.name] = func
        if app:
            for c in app.commanders():
                c.k.registerCommand(self.name, shortcut=None, func=func)
        # Inject ivars for plugins_menu.py.
        func.is_command = True
        func.command_name = self.name
        return func

command = Command
.. @+node:ekr.20150531060949.1: *5* comments
# Comments before class

class aClass:
    pass

pass # trailing comment.
    # indented comment.
# Comments before def.

def spam():
    pass
.. @+node:ekr.20150531054341.1: *5* * and ** args
def spam(a, *args, **keys):
    pass

a = (2 ** 3)
b = (4 * 5)
.. @+node:ekr.20150531052201.1: *5* indented arg lists
foo(a,
     b=2)
foo(a,
    b=2)

def spam(
    c=2,
    d=3,
):
    pass
.. @+node:ekr.20150529183213.1: *5* continued args
foo(a,
     b=2 # foo 1
     # foo 2
  )

def spam(a=2,
    b=3, # spam 1
):
    pass
.. @+node:ekr.20150530182531.1: *5* comments after inner def
def spam(a=2,
    b=3, # spam 1
):

    def inner():
        pass
    # spam 2:

    pass # spam 3.
    # The dedent happens after spam3, so spam2 is over-indented.
.. @+node:ekr.20150530084744.1: *5* doc parts
def f():
    pass
@ The following could be added to the 'else' clause::
    # Accumulate everything else.
.. @+node:ekr.20150528113231.1: *5* indent after inner class
# Indent after inner class.
if QtGui:

    class aClass:
        pass

if Qsci:
    pass
.. @+node:ekr.20150528095649.1: *5* multiple dedent tokens
def get_rc_file(self):
    while True:
        if False:
            return
    foo('oops')
    return
.. @+node:ekr.20150528092108.1: *5* blank lines between classes and defs
class aClass:

    def outer(a=2, b=3):
        a = 2

        def inner():
            b = 2

        c = 3

    class innerClass:
        pass
.. @+node:ekr.20150528093928.1: *5* decorators
class aClass:

    @decr('a')
    def outer():

        @decr('b')
        def inner():
            pass

        pass

    @decr('c')
    class innerClass:
        pass
.. @+node:ekr.20150528060217.1: *5* Inner defs
def outer():
    a = 2

    def inner():
        b = 2

    c = 3
.. @+node:ekr.20150527184101.1: *5* comments
def f1():
    # Comment1, flush.
    line1 = 1
        # Comment2, hanging.
    line2 = 2
    line3 = 3 # Trailing comment.
        # Final hanging comment.
.. @+node:ekr.20150526122041.1: *5* operators
a = -b
c = -(d + e)
b = not (a and b)
c = e or f
.. @+node:ekr.20150527201950.1: *5* strings
s = 'abc \
xyz'
s2 = 'part1' 'part2'
s3 = 'a' + 'b'
.. @+node:ekr.20150527182228.1: *5* Removing blank lines
def f1():
    line1 = 1
    line2 = 2
    line3 = 3

class aClass():
    line4 = 4
    line5 = 5
.. @+node:ekr.20161022040621.1: *3* re: #325: Simplify the organization of commands
@language python
.. @+node:ekr.20161022034842.1: *4* Script: make list of commander attributes that must be preserved
g.cls()
attrs = dir(c)
print('attrs = [')
for attr in attrs:
    print("    '%s'," % attr)
print(']')

.. @+node:ekr.20161022041103.1: *4* @test #325 preserves commander attributes
<< define old_attrs >>
for attr in old_attrs:
    assert hasattr(c, attr), attr

.. @+node:ekr.20161022041138.1: *5* << define old_attrs >>
old_attrs = [
    'BeginUpdate',
    'BringToFront',
    'EndUpdate',
    '_Commands__get_p',
    # '__class__',
    # '__delattr__',
    # '__dict__',
    # '__dir__',
    # '__doc__',
    # '__eq__',
    # '__format__',
    # '__ge__',
    # '__getattribute__',
    # '__gt__',
    # '__hash__',
    # '__init__',
    # '__le__',
    # '__lt__',
    # '__module__',
    # '__ne__',
    # '__new__',
    # '__reduce__',
    # '__reduce_ex__',
    # '__repr__',
    # '__setattr__',
    # '__sizeof__',
    # '__str__',
    # '__subclasshook__',
    # '__weakref__',
    # '_currentCount',
    # '_currentPosition',
    # '_prev_next',
    # '_rootCount',
    # '_style_deltas',
    # '_topPosition',
    'abbrevCommands',
    'abbrev_place_end',
    'abbrev_place_start',
    'abbrev_subst_end',
    'abbrev_subst_env',
    'abbrev_subst_start',
    'about',
    'active_stylesheet',
    'addComments',
    'add_command',
    'alert',
    'allNodes_iter',
    'all_nodes',
    'all_positions',
    'all_positions_iter',
    'all_positions_with_unique_tnodes_iter',
    'all_positions_with_unique_vnodes_iter',
    'all_tnodes_iter',
    'all_unique_nodes',
    'all_unique_positions',
    'all_unique_tnodes_iter',
    'all_unique_vnodes_iter',
    'all_vnodes_iter',
    'allow_at_in_paragraphs',
    'appendStringToBody',
    'atFileCommands',
    'at_root_bodies_start_in_doc_mode',
    'autoindent_in_nocolor',
    'backlinkController',
    'backup',
    'beginUpdate',
    'bigTextController',
    'bodyInitedDict',
    'bodyWantsFocus',
    'bodyWantsFocusNow',
    'bringToFront',
    'cacheListFileName',
    'cacher',
    'canClone',
    'canContractAllHeadlines',
    'canContractAllSubheads',
    'canContractParent',
    'canContractSubheads',
    'canCutOutline',
    'canDehoist',
    'canDeleteHeadline',
    'canDemote',
    'canExpandAllHeadlines',
    'canExpandAllSubheads',
    'canExpandSubheads',
    'canExtract',
    'canExtractSection',
    'canExtractSectionNames',
    'canFindMatchingBracket',
    'canGoToNextDirtyHeadline',
    'canGoToNextMarkedHeadline',
    'canHoist',
    'canMarkChangedHeadlines',
    'canMarkChangedRoots',
    'canMoveOutlineDown',
    'canMoveOutlineLeft',
    'canMoveOutlineRight',
    'canMoveOutlineUp',
    'canPasteOutline',
    'canPromote',
    'canRedo',
    'canSelectThreadBack',
    'canSelectThreadNext',
    'canSelectVisBack',
    'canSelectVisNext',
    'canShiftBodyLeft',
    'canShiftBodyRight',
    'canSortChildren',
    'canSortSiblings',
    'canUndo',
    'canUnmarkAll',
    'cantMoveMessage',
    'changed',
    'chapterCommands',
    'chapterController',
    'checkAllPythonCode',
    'checkBatchOperationsList',
    'checkDrag',
    'checkFileTimeStamp',
    'checkGnxs',
    'checkLinks',
    'checkMoveWithParentWithWarning',
    'checkOutline',
    'checkParentAndChildren',
    'checkPythonCode',
    'checkPythonNode',
    'checkSiblings',
    'checkThreadLinks',
    'check_event',
    'cleanRecentFiles',
    'clearAllHoists',
    'clearAllMarked',
    'clearAllVisited',
    'clearMarked',
    'clearRecentFiles',
    'cleo',
    'clone',
    'cloneFindAllFlattenedMarked',
    'cloneFindAllMarked',
    'cloneFindByPredicate',
    'cloneFindMarkedHelper',
    'cloneFindParents',
    'cloneMarked',
    'cloneToAtSpot',
    'cloneToLastNode',
    'close',
    'collapse_nodes_after_move',
    'collapse_on_lt_arrow',
    'colorPanel',
    'command_count',
    'commandsDict',
    'computeCopiedBunchList',
    'computeVnodeInfoDict',
    'computeWindowTitle',
    'config',
    'configInited',
    'contractAllHeadlines',
    'contractAllOtherNodes',
    'contractIfNotCurrent',
    'contractNode',
    'contractNodeOrGoToParent',
    'contractParent',
    'contractSubtree',
    'contractVisitedNodes',
    'controlCommands',
    'convertAllBlanks',
    'convertAllTabs',
    'convertBlanks',
    'convertCommands',
    'convertTabs',
    'copyMarked',
    'copyOutline',
    'createCloneFindPredicateRoot',
    'createCommandNames',
    'createLastChildNode',
    'createMoveMarkedNode',
    'createMyLeoSettings',
    'createNodeFromExternalFile',
    'createNodeHierarchy',
    'create_nonexistent_directories',
    'currentPosition',
    'currentPositionHasNext',
    'currentPositionIsRootPosition',
    'currentVnode',
    'cutOutline',
    'db',
    'debugCommands',
    'dedentBody',
    'dehoist',
    'deleteComments',
    'deleteMarked',
    'deleteOutline',
    'deletePositionsInList',
    'demote',
    'disableCommandsMessage',
    'doBatchOperations',
    'doCommand',
    'doubleClickFlag',
    'dragAfter',
    'dragCloneAfter',
    'dragCloneToNthChildOf',
    'dragToNthChildOf',
    'dumpOutline',
    'dumpPosition',
    'editCommands',
    'editFileCommands',
    'editHeadline',
    'edit_widget',
    'enableMenuBar',
    'endEditing',
    'endUpdate',
    'endsParagraph',
    'executeMinibufferCommand',
    'executeScript',
    'executeScriptHelper',
    'exists',
    'expandAllAncestors',
    'expandAllHeadlines',
    'expandAllSubheads',
    'expandLevel1',
    'expandLevel2',
    'expandLevel3',
    'expandLevel4',
    'expandLevel5',
    'expandLevel6',
    'expandLevel7',
    'expandLevel8',
    'expandLevel9',
    'expandNextLevel',
    'expandNode',
    'expandNodeAndGoToFirstChild',
    'expandNodeOrGoToFirstChild',
    'expandOnlyAncestorsOfNode',
    'expandPrevLevel',
    'expandSubtree',
    'expandToLevel',
    'expansionLevel',
    'expansionNode',
    'exportHeadlines',
    'extract',
    'extractDef',
    'extractPythonMethod',
    'extractRef',
    'extractSection',
    'extractSectionNames',
    'fileCommands',
    'fileName',
    'findBoundParagraph',
    'findCommands',
    'findMatchingBracket',
    'findNextClone',
    'findNodeOutsideAnyAtFileTree',
    'findRootPosition',
    'findSectionName',
    'find_b',
    'find_h',
    'finishCreate',
    'firstVisible',
    'fixed',
    'fixedWindowPosition',
    'fixedWindowPositionData',
    'flattenOutline',
    'flattenOutlineToNode',
    'focus_border_color',
    'focus_border_command_state_color',
    'focus_border_overwrite_state_color',
    'focus_border_width',
    'fontPanel',
    'forceExecuteEntireBody',
    'force_redraw',
    'frame',
    'free_layout',
    'fullCheckOutline',
    'getBodyLines',
    'getLanguageAtCursor',
    'getNodeFileName',
    'getNodePath',
    'getSelectedPositions',
    'getTabWidth',
    'getTime',
    'get_focus',
    'get_requested_focus',
    'goNextVisitedNode',
    'goPrevVisitedNode',
    'goToFirstNode',
    'goToFirstSibling',
    'goToFirstVisibleNode',
    'goToLastNode',
    'goToLastSibling',
    'goToLastVisibleNode',
    'goToLineNumber',
    'goToNextClone',
    'goToNextDirtyHeadline',
    'goToNextHistory',
    'goToNextMarkedHeadline',
    'goToNextSibling',
    'goToParent',
    'goToPrevHistory',
    'goToPrevSibling',
    'goToScriptLineNumber',
    'gotoCommands',
    'gui',
    'hasAmbiguousLanguage',
    'hash',
    'helpCommands',
    'hiddenRootNode',
    'hideInvisibles',
    'hoist',
    'hoistStack',
    'hookFunction',
    'idle_focus_count',
    'idle_focus_helper',
    'ignoreChangedPaths',
    'ignored_at_file_nodes',
    'importAnyFile',
    'importAtFile',
    'importAtRoot',
    'importCWEBFiles',
    'importCommands',
    'importDerivedFile',
    'importFlattenedOutline',
    'importMOREFiles',
    'importNowebFiles',
    'importTabFiles',
    'import_error_nodes',
    'inCommand',
    'in_qt_dialog',
    'incrementalRecolorFlag',
    'indentBody',
    'initAfterLoad',
    'initCommandIvars',
    'initConfigSettings',
    'initDebugIvars',
    'initDocumentIvars',
    'initEventIvars',
    'initFileIvars',
    'initObjectIvars',
    'initObjects',
    'initOptionsIvars',
    'initSettings',
    'init_error_dialogs',
    'initialFocusHelper',
    'insertBodyTime',
    'insertChild',
    'insertHeadline',
    'insertHeadlineBefore',
    'invalidateFocus',
    'ipythonController',
    'isChanged',
    'isCurrentPosition',
    'isRootPosition',
    'isZipped',
    'is_unusual_focus',
    'k',
    'keyHandler',
    'keyHandlerCommands',
    'killBufferCommands',
    'lastTopLevel',
    'lastVisible',
    'last_dir',
    'last_no_focus',
    'last_unusual_focus',
    'leoCommands',
    'leoDocumentation',
    'leoHome',
    'leoQuickStart',
    'leoTestManager',
    'loading',
    'logWantsFocus',
    'logWantsFocusNow',
    'looksLikeDerivedFile',
    'mFileName',
    'mRelativeFileName',
    # 'macroCommands',
    'make_node_conflicts_node',
    'markAllAtFileNodesDirty',
    'markAtFileNodesDirty',
    'markChangedHeadlines',
    'markChangedRoots',
    'markHeadline',
    'markSubheads',
    'max_pre_loaded_body_chars',
    'menuAccels',
    'miniBufferWidget',
    'minibufferWantsFocus',
    'minibufferWantsFocusNow',
    'moveMarked',
    'moveOutlineDown',
    'moveOutlineLeft',
    'moveOutlineRight',
    'moveOutlineUp',
    'navHelper',
    'navPrefix',
    'navQuickKey',
    'navTime',
    'new',
    'nodeConflictFileName',
    'nodeConflictList',
    'nodeHistory',
    'notValidInBatchMode',
    'nullPosition',
    'onCanvasKey',
    'open',
    'openCheatSheet',
    'openCompareWindow',
    'openDirectory',
    'openLeoPlugins',
    'openLeoPy',
    'openLeoScripts',
    'openLeoSettings',
    'openLeoTOC',
    'openLeoTutorials',
    'openLeoUsersGuide',
    'openLeoVideos',
    'openMyLeoSettings',
    'openPythonWindow',
    'openRecentFile',
    'openUnittest',
    'openWith',
    'os_path_finalize',
    'os_path_finalize_join',
    'outerUpdate',
    'outlineHasInitialFocus',
    'outlineToCWEB',
    'outlineToNoweb',
    'outlineToNowebDefaultFileName',
    'output_doc_flag',
    'output_initial_comment',
    'output_newline',
    'p',
    'page_width',
    'pasteOutline',
    'pasteOutlineRetainingClones',
    'persistenceController',
    'pluginsMenu',
    'positionExists',
    'preferences',
    'printCommandsDict',
    'printingController',
    'promote',
    'promptingForClose',
    'putBitsFlag',
    'putHelpFor',
    'queryReplaceCommands',
    'raise_error_dialogs',
    'readAtAutoNodes',
    'readAtFileNodes',
    'readAtShadowNodes',
    'readFileIntoNode',
    'readOutlineOnly',
    'read_only',
    'recolor',
    'recolor_now',
    'recreateGnxDict',
    'rectangleCommands',
    'recursiveImport',
    'redirectScriptOutput',
    'redirect_execute_script_output_to_log_pane',
    'redraw',
    'redrawAndEdit',
    'redraw_after_contract',
    'redraw_after_expand',
    'redraw_after_head_changed',
    'redraw_after_icons_changed',
    'redraw_after_select',
    'redraw_now',
    'reformatBody',
    'reformatParagraph',
    'refreshFromDisk',
    'relativeFileName',
    'relative_path_base_directory',
    'removeSentinels',
    'remove_sentinels_extension',
    'requestBringToFront',
    'requestCloseWindow',
    'requestRecolor',
    'requestRecolorFlag',
    'requestRedrawFlag',
    'request_focus',
    'requestedFocusWidget',
    'requestedIconify',
    'revert',
    'rootPosition',
    'rootVnode',
    'rp_get_args',
    'rp_get_leading_ws',
    'rp_reformat',
    'rp_wrap_all_lines',
    'rstCommands',
    'safe_all_positions',
    'save',
    'saveAll',
    'saveAs',
    'saveAsUnzipped',
    'saveAsZipped',
    'saveAsZippedHelper',
    'saveTo',
    'save_clears_undo_buffer',
    'scanAllDirectives',
    'scanAtPathDirectives',
    'scanAtPathDirectivesCount',
    'scanAtRootDirectives',
    'searchCommands',
    'selectAtSettingsNode',
    'selectPosition',
    'selectThreadBack',
    'selectThreadNext',
    'selectVisBack',
    'selectVisNext',
    'selectVnode',
    'setBodyString',
    'setChanged',
    'setCloneFindByPredicateIcon',
    'setComplexCommand',
    'setCurrentDirectoryFromContext',
    'setCurrentPosition',
    'setCurrentVnode',
    'setFileTimeStamp',
    'setHeadString',
    'setLog',
    'setMarked',
    'setPositionAfterSort',
    'setRootPosition',
    'setRootVnode',
    'setTopPosition',
    'setTopVnode',
    'setWindowPosition',
    'set_focus',
    'shadowController',
    'shortFileName',
    'shortFilename',
    'shouldBeExpanded',
    'showInvisibles',
    'showInvisiblesHelper',
    'singleLineParagraph',
    'smart_tab',
    'sortChildren',
    'sortRecentFiles',
    'sortSiblings',
    'sparce_spell',
    'sparse_find',
    'sparse_move',
    'sparse_spell',
    'spellCommands',
    'startRedrawCount',
    'startsParagraph',
    'status_line_unl_mode',
    'stayInTreeAfterSelect',
    'styleSheetManager',
    'stylesheet',
    'suppressHeadChanged',
    'syntaxErrorDialog',
    'tabNannyNode',
    'tab_width',
    'tangle',
    'tangleAll',
    'tangleCommands',
    'tangleMarked',
    'tangle_batch_flag',
    'tangle_errors',
    'target_language',
    'testManager',
    'theScriptingController',
    'theTagController',
    'toggleAngleBrackets',
    'toggleShowInvisibles',
    'toggleSparseMove',
    'topPosition',
    'topVnode',
    'traceFocus',
    'trace_focus_count',
    'trace_idle_focus',
    'trailing_body_newlines',
    'treeFocusHelper',
    'treeSelectHelper',
    'treeWantsFocus',
    'treeWantsFocusNow',
    'trimTrailingLines',
    'undo_granularity',
    'undoer',
    'unformatParagraph',
    'universalCallback',
    'universallCallback',
    'unmarkAll',
    'unredirectScriptOutput',
    'unreformat',
    'untangle',
    'untangleAll',
    'untangleMarked',
    'untangle_batch_flag',
    'updateBodyPane',
    'updateSyntaxColorer',
    'use_body_focus_border',
    'use_focus_border',
    'use_header_flag',
    'use_plugins',
    'user_dict',
    'validateOutline',
    'verbose_check_outline',
    'vimCommands',
    'vim_mode',
    'visLimit',
    'vnode2allPositions',
    'vnode2position',
    'weave',
    'widgetWantsFocus',
    'widgetWantsFocusNow',
    'widget_name',
    'wrappedFileName',
    'writeFileFromNode',
    'writeScriptFile',
    'write_script_file',
    'write_strips_blank_lines',
]
.. @+node:ekr.20161120163831.1: *3* @test new generators
g.cls()
import imp
import leo.core.leoCommands as leoCommands
import leo.core.leoNodes as leoNodes
imp.reload(leoCommands)
imp.reload(leoNodes)
root = g.findTopLevelNode(c, 'Code')
for p in root.nearest_unique_roots():
    print(p.h)
print('='*30)
for p in c.all_unique_roots():
    print(p.h)
.. @+node:ekr.20170120111046.1: *3* ReportTraverser
.. @+node:ekr.20170120111046.2: *4* @test rt
<< imports >>
g.cls()
if c.isChanged(): c.save()
@others
path = g.os_path_finalize_join(g.app.loadDir,
    '..', 'test', 'report_test.py') # 'leoAst.py')
assert g.os_path_exists(path), path
fn = g.shortFileName(path)
source = open(path, 'r').read()
node = ast.parse(source, filename=fn, mode='exec')
s = HTMLReportTraverser(debug=True).main(fn, node)
out_fn = g.os_path_finalize_join(g.app.loadDir, '..', 'test', 
    'HTMLReportTraverser_test.html')
open(out_fn, 'w').write(s)
# print('%s %s' % (len(s), out_fn))
os.startfile(out_fn)
# os.system('ed ' + out_fn)
.. @+node:ekr.20170120111046.3: *5* << imports >>
import leo.core.leoAst as leoAst
import ast
import os
import xml.sax.saxutils as saxutils
import textwrap
# Needed only for the script version
AstFullTraverser = leoAst.AstFullTraverser
.. @+node:ekr.20170120111046.4: *4* @@file ../test/report_test.py
'''Test file for HTMLReportTraverser class.'''
<< includes >>
@others

.. @+node:ekr.20170120111046.5: *5* << includes >>
import ast
import xml.sax.saxutils as saxutils
import leo.core.leoGlobals as g
.. @+node:ekr.20170120111046.6: *5* class TestClass
class TestClass(object):
    '''A class exercising important cases in the HTMLReportTraverser code.'''
    @others
.. @+node:ekr.20170120111046.7: *6* if_tests
def if_tests(self, a, b):
    '''Test if statements, especially 'elif' vs. 'else if'.'''
    if a and b:
        rt.gen("<div class='%s' %s>" % (full_class_name, extra))
    elif a:
        rt.gen("<div class='%s'>" % (full_class_name))
    else:
        assert not extra
        rt.gen("<div>")
    if a:
        print('a')
    else: # elif is not correct here.
        if b:
            print(b)
        print(c)
.. @+node:ekr.20170120111046.8: *6* comprehension_tests
def comprehension_tests(self):
    '''Test comprehensions'''
    for i, aList in enumerate(self.line_tokens):
        print('hi')
    return [z for z in aList if self.token_kind(z) == 'string']
.. @+node:ekr.20170120111046.9: *6* try_tests
def try_tests(self):
    '''Test try/except/finally'''
    try:
        for z in node.optional_vars:
            vars_list.append(self.visit(z))
    except TypeError:
        vars_list.append(self.visit(node.optional_vars))
    finally:
        print('ooops')
        raise
    
.. @+node:ekr.20161117164321.1: *3* Unused unit tests
# Some of these tests give warnings.
.. @+node:ekr.20161117164510.1: *4* Tk gui tests
.. @+node:ekr.20161117164510.2: *5* @test leoBody is subset of leoTkBody
if g.app.gui.guiName() == 'tkinter':

    pc = g.app.pluginsController
    tkGui = pc.loadOnePlugin('leo.plugins.tkGui',verbose=False)
    assert(tkGui)

    import leo.core.leoFrame as leoFrame
    import inspect,sys

    baseClass = leoFrame.leoBody
    subClasses  = (tkGui.leoTkinterBody,leoFrame.nullBody)
    baseObject = c.frame.body

    methods = inspect.getmembers(baseClass,inspect.ismethod)
    methodNames = [z[0] for z in methods]

    for name in baseObject.mustBeDefinedOnlyInBaseClass:
        try:
            assert name in methodNames, 'not defined in base class %s.%s' % (baseClass.__name__,name)
        except AssertionError:
            exctype, value = sys.exc_info()[:2]
            print(value)
            raise

    for subClass in subClasses:
        subclassName = subClass.__name__
        for name in methodNames:
            base_func = getattr(baseClass,name)
            sub_func =  getattr(subClass,name)
            try:
                if name in baseObject.mustBeDefinedOnlyInBaseClass:
                    assert base_func.im_func == sub_func.im_func, 'defined in subclass %s.%s' % (subclassName,name)
                if name in baseObject.mustBeDefinedInSubclasses:
                    assert base_func.im_func != sub_func.im_func, 'not defined in subclass %s.%s' % (subclassName,name)
            except AssertionError:
                #raise
                exctype, value = sys.exc_info()[:2]
                print(value)
.. @+node:ekr.20161117164510.3: *5* @test leoFrame is subset of leoTkFrame
if g.app.gui.guiName() == 'tkinter':

    pc = g.app.pluginsController
    tkGui = pc.loadOnePlugin('leo.plugins.tkGui',verbose=False)

    import leo.core.leoFrame as leoFrame
    import inspect

    baseClass = leoFrame.leoFrame
    subClasses  = (tkGui.leoTkinterFrame,leoFrame.NullFrame)
    baseObject = c.frame

    methods = inspect.getmembers(baseClass,inspect.ismethod)
    methodNames = [z[0] for z in methods]

    for name in baseObject.mustBeDefinedOnlyInBaseClass:
        assert name in methodNames, 'not defined in base class %s.%s' % (baseClass.__name__,name)

    for subClass in subClasses:
        subclassName = subClass.__name__
        for name in methodNames:
            base_func = getattr(baseClass,name)
            sub_func =  getattr(subClass,name)
            if name in baseObject.mustBeDefinedOnlyInBaseClass:
                assert base_func.im_func == sub_func.im_func, 'defined in subclass %s.%s' % (subclassName,name)
            if name in baseObject.mustBeDefinedInSubclasses:
                assert base_func.im_func != sub_func.im_func, 'not defined in subclass %s.%s' % (subclassName,name)
.. @+node:ekr.20161117164510.4: *5* @test leoGui is subset of leoTkGui
if g.app.gui.guiName() == 'tkinter':

    pc = g.app.pluginsController
    tkGui = pc.loadOnePlugin('leo.plugins.tkGui',verbose=False)

    import leo.core.leoGui as leoGui
    import inspect

    baseClass = leoGui.leoGui
    subClasses  = (tkGui.tkinterGui,) # nullGui can inherit almost all leoGui dummy methods.
    baseObject = g.app.gui

    methods = inspect.getmembers(baseClass,inspect.ismethod)
    methodNames = [z[0] for z in methods]

    for name in baseObject.mustBeDefinedOnlyInBaseClass:
        assert name in methodNames, 'not defined in base class %s.%s' % (baseClass.__name__,name)

    for subClass in subClasses:
        subclassName = subClass.__name__
        for name in methodNames:
            base_func = getattr(baseClass,name)
            sub_func =  getattr(subClass,name)
            try:
                if name in baseObject.mustBeDefinedOnlyInBaseClass:
                    assert base_func.im_func == sub_func.im_func, 'defined in subclass %s.%s' % (subclassName,name)
                if name in baseObject.mustBeDefinedInSubclasses:
                    assert base_func.im_func != sub_func.im_func, 'not defined in subclass %s.%s' % (subclassName,name)
            except AssertionError:
                raise
.. @+node:ekr.20161117164510.5: *5* @test leoTree is subset of leoTkTree
if g.app.gui.guiName() == 'tkinter':

    pc = g.app.pluginsController
    tkGui = pc.loadOnePlugin('leo.plugins.tkGui',verbose=False)

    import leo.core.leoFrame as leoFrame
    import inspect

    baseClass = leoFrame.leoTree
    subClasses  = (tkGui.leoTkinterTree,leoFrame.nullTree)
    baseObject = c.frame.tree

    methods = inspect.getmembers(baseClass,inspect.ismethod)
    methodNames = [z[0] for z in methods]

    for name in baseObject.mustBeDefinedOnlyInBaseClass:
        assert name in methodNames, 'not defined in base class %s.%s' % (baseClass.__name__,name)

    for subClass in subClasses:
        subclassName = subClass.__name__
        for name in methodNames:
            base_func = getattr(baseClass,name)
            sub_func =  getattr(subClass,name)
            if name in baseObject.mustBeDefinedOnlyInBaseClass:
                assert base_func.im_func == sub_func.im_func, 'defined in subclass %s.%s' % (subclassName,name)
            if name in baseObject.mustBeDefinedInSubclasses:
                assert base_func.im_func != sub_func.im_func, 'not defined in subclass %s.%s' % (subclassName,name)
.. @+node:ekr.20161117164321.3: *4* @@@test BaseTextWrapper methods
import leo.core.leoFrame as leoFrame

w = leoFrame.BaseTextWrapper(c,'base-name','class-name',widget=None)

def check(expected):
    s = w.getAllText()
    assert s == expected,'expected %s got %s' % (expected,s)

w.setAllText('')        ; check('')
w.appendText('abc')     ; check('abc')
w.delete(1,2)           ; check('ac')
w.insert(0,'xy')        ; check('xyac')
w.insert(1,'z')         ; check('xzyac')
    # w.replace(2,4,'ABCD') ; check('xzABCDc')
        # This method no longer exists.  It is not used anywhere.
    # w.setSelectionRange(3,6)
    # s = w.getSelectedText()
    # assert s == 'BCD',repr(s)

w.deleteTextSelection() ; check('xzyac')
.. @+node:ekr.20161117164321.4: *4* @@@test writing a .leo file retains orphan bits
# Not valid for external tests: uses @<file> node.
if not g.app.isExternalUnitTest:
    
    # Writing a .leo file must retain orphan bits of erroneous external files.
    
    h = '@file nonexistent-directory/orphan-bit-test.txt'
    p2 = g.findNodeAnywhere(c,h)
    assert p2,'not found: %s' % (h)
    assert p2.isOrphan(),'not an orphan originally'
    
    # It's dangerous to do the write, but this does test the bug fix.
    c.atFileCommands.clearAllOrphanBits(p2)
    assert p2.isOrphan(),'not an orphan after calling at.clearAllOrphanBits'
.. @+node:ekr.20161117164321.5: *4* @@test ptb.comment_leo_lines & ptb.uncomment_leo_lines
import leo.core.leoBeautify as leoBeautify
for p in p.children():
    b = leoBeautify.PythonTokenBeautifier(c)
    s1 = p.b
    comment,s = b.comment_leo_lines(p)
    s2 = b.uncomment_leo_lines(comment,p,s)
    assert s1 == s2,('\ns...\n%s\ns2...\n%s' % (s,s2))
.. @+node:ekr.20161117164321.6: *5* indented docstring
def f ():
    pass
@ Line 1
    Line 2
.. @+node:ekr.20161117164321.7: *5* indented docstring2
def f(): 
    pass
@ The following could be added to the 'else' clause::
    # Accumulate everything else.
.. @+node:ekr.20161117164321.8: *4* @ignore @shadow tests
.. @+node:ekr.20161117164321.9: *5* @@test at.readOneAtShadowNode retains @shadow links clones
# Important: the child of this node must be a clone of
# the corresponding node in @shadow unittest/at-shadow-unlink-clones.py

# The @shadow node will not exist for an external test.
if not g.app.isExternalUnitTest:
    try:
        # print('start',p.h)
        b = c.undoer.beforeChangeTree(p)
        h = '@shadow unittest/at-shadow-unlink-clones.py'
        root = g.findNodeAnywhere(c,h)
        assert root
        assert root.h == h,repr(root.h)
        child = p.firstChild()
        assert child
        assert child.isCloned(),'fail 1: test not set up properly'
        c.selectPosition(root)
        fn = root.atShadowFileNodeName()
        assert fn
        c.atFileCommands.readOneAtShadowNode (fn,root,force=True)
        c.undoer.afterChangeTree(p,'fc.readOneAtShadowNode',b)
        assert child.isCloned(),'fail 2: intended test fails'
        c.undoer.undo()
    finally:
        c.selectPosition(p)
        c.redraw()
.. @+node:ekr.20161117164321.10: *6* Node 1
# node 1 text A.
.. @+node:ekr.20161117164321.11: *5* @@test goto-global-line @shadow
# Not valid for external tests: uses @<file> node.
if not g.app.isExternalUnitTest:

    h = '@shadow unittest/at-shadow-line-number-test.py'
    root1 = g.findNodeAnywhere(c,h)
    assert root1
    assert root1.isAnyAtFileNode()
    
    fileName,lines,n,root2 = c.GoToLineNumber(c).setup_file(n=6,p=root1)
    assert fileName == h[8:],'fileName'
    assert root2 == root1
    
    if 0:
        print('root:%s, isRaw:%s, n:%s, len(lines): %s' % (
            root and root.h,isRaw,n,len(lines)))
.. @+node:ekr.20161117164321.12: *4* @ignore from leoPy.leo
import leo.core.leoImport as leoImport
ic = c.importCommands
hs = leoImport.HtmlScanner(importCommands=ic,atAuto=True)

s1 = '''
<table id="1"> <table id="2">
<contents/>
</table>
</table>
'''

s2 = '''
<table id="1"> 
<table id="2">
<contents/>
</table>
</table>
'''

t1 = 
assert result == expected,'expected...\n%s\ngot...\n%s' % (
    repr(expected),repr(result))
.. @+node:ekr.20161117164321.13: *5* @test html string
s = '''\
<HTML>
<head>
    <title>Bodystring</title>
</head>
<body class='bodystring'>
<div id='bodydisplay'></div>
</body>
</html>
'''

html_tags = ('body','head','html','table',) # 'div',
setting = 'import_html_tags'

# Settings now work when run externally.
c.config.set(setting,'data',html_tags)
tags = c.config.getData(setting)
assert tags == html_tags,len(tags)

g.app.unitTestDict ['expectedErrors'] = 0

showTree = True

c.importCommands.htmlUnitTest(p,s=s,showTree=showTree)

if showTree:
    # g.cls()
    for p in p.subtree():
        print('\n***** %s\n' %p.h)
        print(p.b)
.. @+node:ekr.20161117164321.14: *6* @file c:/leo.repo/trunk/leo/core/html string
@language xml
@tabwidth -4
@others

.. @+node:ekr.20161117164321.23: *7* html
<HTML>
@others
</html>
.. @+node:ekr.20161117164321.24: *8* head

<head>
    <title>Bodystring</title>
</head>
.. @+node:ekr.20161117164321.25: *8* body
<body class='bodystring'>
<div id='bodydisplay'></div>
</body>
.. @+node:ekr.20161117164321.18: *6* @file c:/leo.repo/trunk/leo/core/html string
@language xml
@tabwidth -4
@others

.. @+node:ekr.20161117164321.23: *7* html
<HTML>
@others
</html>
.. @+node:ekr.20161117164321.24: *8* head

<head>
    <title>Bodystring</title>
</head>
.. @+node:ekr.20161117164321.25: *8* body
<body class='bodystring'>
<div id='bodydisplay'></div>
</body>
.. @+node:ekr.20161117164321.22: *6* @file c:/leo.repo/trunk/leo/core/html string
@language xml
@tabwidth -4
@others

.. @+node:ekr.20161117164321.23: *7* html
<HTML>
@others
</html>
.. @+node:ekr.20161117164321.24: *8* head

<head>
    <title>Bodystring</title>
</head>
.. @+node:ekr.20161117164321.25: *8* body
<body class='bodystring'>
<div id='bodydisplay'></div>
</body>
.. @+node:ekr.20161117164321.26: *5* @test HtmlScanner.filterTokens
import leo.core.leoImport as leoImport
ic = c.importCommands
hs = leoImport.HtmlScanner(importCommands=ic,atAuto=True)
strip = hs.stripTokens
dump  = hs.formatTokens

s1 = '''<table id="1"><table id="2">
<contents/>
</table>
</table>'''

s2 = '<table id="1"><table id="2"><contents/></table></table>'
    
t1 = hs.tokenize(s1)
t2 = hs.tokenize(s2)
f1 = hs.filterTokens(t1)
f2 = hs.filterTokens(t2)

assert strip(f1) == strip(f2),'f1...\n%s\nf2...\n%s' % (
    dump(f1),dump(f2))
    
if 0:
    print(dump(f1))
.. @+node:ekr.20161117164321.27: *5* @test import dataN.html
fn = r'c:\recent\data.html'

# fn = r'c:\recent\data-smaller.html'
# fn = r'c:\recent\data666.html'

# These all pass on data.html:
    # html_tags = ('html','head','body',)
    # html_tags = ('html','head','body','table',)
    # html_tags = ('html','head','body','table','div',)
    # html_tags = ('html','head','body','table','div','script',)
    # html_tags = ('html','head','body','table','div','script','link',)
    # html_tags = ('html','head','body','table','div','script','link','p',)

html_tags = ('html','head','body','table','div','script','p','td','tr',)

# Settings now work when run externally.
setting = 'import_html_tags'
c.config.set(setting,'data',html_tags)
tags = c.config.getData(setting)
assert tags == html_tags,len(tags)

g.cls()

c.importCommands.importFilesCommand(files=[fn], treeType='@file')
.. @+node:ekr.20161117164321.28: *5* @test unicode stuff
@first # -*- coding: utf-8 -*-

table = (
    'test',
    'Ä 궯 奠',
    'Ä 궯 奠 after', # fails with cp6501: after is duplicated.
)

print('*'*20)
print('isPython3: %s' % g.isPython3)

for s in table:
    if g.isPython3:
        s = s.encode('ascii','replace') # create bytes.
    g.es(repr(s))
    g.es(s)
    g.pr ('g.pr(s)       : %s' % s)
    g.pr ('g.pr(repr(s)) : %s' % repr(s))
    print('print(s)      : %s' % s)
    print('print(repr(s)): %s' % s)
.. @+node:ekr.20161117164321.29: *5* @test external text operations
assert g.app.isExternalUnitTest

body = c.frame.body
assert repr(body.widget).startswith('stringTextWidget')
assert body.widget == body.bodyCtrl

w = body.bodyCtrl
w.setAllText(p.b)
assert p.b == w.getAllText()
.. @+node:ekr.20161117164321.30: *5* @test nullBody text operations
# print('isExternalUnitTest',g.app.isExternalUnitTest)

if g.app.isExternalUnitTest:
    body = c.frame.body
else:
    import leo.core.leoCommands as leoCommands
    import leo.core.leoFrame as leoFrame
    import leo.core.leoGui as leoGui
    
    # Important: external unit tests should execute in this environment.
    nullGui   = leoGui.NullGui('null gui')
    nullFrame = leoFrame.NullFrame(title='nullFrame title',gui=nullGui)
    c2 = leoCommands.Commands(nullFrame,fileName='<empty fileName>')
    nullFrame.c = c2
    body = leoFrame.nullBody(frame=nullFrame,parentFrame=None)
    assert repr(body).startswith('<leo.core.leoFrame.nullBody')

# Now test some basic operations.
assert repr(body.widget).startswith('stringTextWidget')
assert body.widget == body.bodyCtrl
w = body.bodyCtrl

w.setAllText(p.b)
assert p.b == w.getAllText()
.. @+node:ekr.20161117164321.31: *5* @test g.python_tokenize
# h = 'g.python_tokenize'
# p = p.firstChild()
# assert p.h == h
tokens = g.python_tokenize(p.b,line_numbers=False)

# tokens = [(kind,val) for (kind,val,line_number) in tokens]

# First, the basic check
tokens1 = [val for kind,val in tokens]
s = ''.join(tokens1)
assert p.b == s,repr(s)

if 0:
    for z in tokens:
        kind,val = z
        print('%6s %s' % (kind,repr(val)))
        
# Next, start filtering.
tokens = [(kind,g.choose(kind=='string','"S"',val)) for kind,val in tokens]

if 0: # Delete whitespace.
    tokens = [(kind,val) for (kind,val) in tokens if kind != 'ws']
    tokens = [(kind,g.choose(kind=='id',val+' ',val)) for (kind,val) in tokens]

# Last: stringize.
tokens = [val for kind,val in tokens if kind != 'comment']
# print(''.join(tokens))

if 0: # Print lines containing '='
    s = ''.join(tokens)
    for ch in '()[]{}<>.,:=+-/':
        s = s.replace(' '+ch,ch)
    aList = [z for z in g.splitLines(s)
        if z.find('=') > -1] # and not z.find('+=')>-1 and not z.find('-=')>-1]
    print(''.join(aList))
    
.. @+node:ekr.20161117164321.32: *4* @ignore LeoInspect unit tests
@language python
.. @+node:ekr.20161117164321.33: *5* Basic tests
.. @+node:ekr.20161117164321.34: *6* @test leoInspect with multiple files
import leo.core.leoInspect as leoInspect
import os
import time

<< define old_s >>
<< define s >>

@others

aList = (
    'leoAtFile.py',
    'leoEditCommands.py',
)
test(files=aList,print_stats=False,s=None,print_times=True)
.. @+node:ekr.20161117164321.35: *7* << define old_s >>
# import leo.core.leoGlobals
# import leo.core.leoGlobals as g
# from leo.core.leoGlobals import pr as pr2
# from leo.core.leoGlobals import trace

s_old = '''\
import sys

aGlobal = 5
# aGlobal2 is not explicitly defined.

c = [z for z in 'abc']

def myFunc():
    n1,n2,n3,junk,junk=sys.version_info
    a = self.b
    for z in a:
        print(z)
    with A() as a:
        print(a,b)
        
def test():
    a = b # UnboundLocalError.
    b = 1
    c = 2 # Any def will do at present.
    print(g)
    print(c.frame.body)
    print(c.frame.body.xxx.yyy)
    print(b.yyy) # no check will be made.
    print(xxx.yyy)
    for c in 'abc':
        print(c,b)
        print(g)


class myClass:
    
    def __init__(self,c):
        self.a = True
        self.b = None
        c.frame.xxxx
        
    def spam(self,a,b,c=5,*args,**keys):
        global aGlobal2
        aGlobal2 = 'abc'
        self.a = b
        self.a = x
        
    def no_self(a):
        pass
        
    def test_lambda(self):
        f = lambda a,b: a

    def test_comprehension(self):
        z2 = [z for z in 'abc']
        
aGlobal3 = 4 # This should be defined everywhere.

def test():
    # a = ','.join(['a','b'])
    p = 5
    # print(p.parent().h)
    # print(g.app.windowList[0])
    print(p)
    print(g)

'''

# import leo.core.leoCommands as leoCommands
.. @+node:ekr.20161117164321.36: *7* << define s >>
s = '''\

import leo.core.leoGlobals as g

def test(c):
    a = 5
    f = c.frame
    c.frame.body.bodyCtrl = w

'''

s = g.adjustTripleString(s,-4)
# print(s)
.. @+node:ekr.20161117164321.37: *7* test
def test(files,print_stats=True,s=None,print_times=True):
   
    t1 = time.time()
    sd = leoInspect.SemanticData(controller=None)

    if s: # Use test string.
        fn = '<test file>'
        leoInspect.InspectTraverser(fn,sd).traverse(s)
    else:
        for fn in files:
            print(g.shortFileName(fn))
            s = leoInspect.LeoCoreFiles().get_source(fn)
            if s:
                leoInspect.InspectTraverser(fn,sd).traverse(s)
            else:
                print('file not found: %s' % (fn))
           
    sd.total_time = time.time()-t1
    
    if print_times: sd.print_times()
    if print_stats: sd.print_stats()
.. @+node:ekr.20161117164321.38: *6* @test leoInspect.module.classes
import leo.core.leoInspect as leoInspect

dump_classes = False
print_modules = True
print_functions = True
print_stats = False
print_times = False

# if dump_modules or print_stats:
    # g.cls()

m = leoInspect.module(fn='leoApp.py',sd=None,
    print_stats=print_stats,print_times=print_times)
    
if 0:
    print(m)
    for o in m.classes():
        if dump_classes:
            o.dump()
        if print_modules:
            print(o)
        if print_functions:
            for f in o.functions():
                print('  %s' % f)
.. @+node:ekr.20161117164321.39: *6* @test leoInspect.module.defs
import leo.core.leoInspect as leoInspect

# g.cls()

m = leoInspect.module(fn='leoEditCommands.py')
    #,sd=None,print_stats=False,print_times=False)
    
if 0:
    print(m)
    for z in m.classes():
        print(z)
        for z2 in z.defs():
            name = z2.tree_ptr.name
            aList = z2.call_args_of(name)
            args = ','.join(aList)
            print(' %s(%s)' % (name,args))
.. @+node:ekr.20161117164321.40: *6* @test leoInspect.module.classes 2
import leo.core.leoInspect as leoInspect

# g.cls()

m = leoInspect.module(fn='leoEditCommands.py')
    #,sd=None,print_stats=False,print_times=False)
    
if 0:
    for z in m.classes():
        print(z)
.. @+node:ekr.20161117164321.41: *6* @test leoInspect.module.statements
import leo.core.leoInspect as leoInspect

# g.cls()

m = leoInspect.module(fn='leoEditCommands.py')
    #,sd=None,print_stats=False,print_times=False)

if 0:
    print(m)
    for z in m.classes():
        print(z)
        for z2 in z.defs():
            print(z2)
.. @+node:ekr.20161117164321.42: *6* @test leoInspect (leoEditCommands.py)
import leo.core.leoInspect as leoInspect

# g.cls()

m = leoInspect.module(fn='leoEditCommands.py')

def show(o):
    print('%-5s %s' % (o.line_number(),o.format()))

var = '.widget'
func = 'w.insert'

if 0:
    print('\nAssignments to %s...\n' % (var))
    for o in m.assignments_to(var):
        show(o)
        
    print('\nAssignments using %s...\n' % (var))
    for o in m.assignments_using(var):
        show(o)
        
    print('\nCalls to %s...\n' % (func))
    for o in m.calls_to(func):
        show(o)
        
    if 1:
        classes = m.classes()
        for d in classes[0].defs():
            print('')
            print(d)
            for z in d.statements():
                # print(z.tree())
                # print(z.sd.dump_ast(z.tree()))
                lines = g.splitLines(z.format())
                for line in lines:
                    print('  %s' % (line))
.. @+node:ekr.20161117164321.43: *6* @test leoInspect.module (s)
import leo.core.leoInspect as leoInspect

def show(o,indent=0):
    # print('\n%s\n' % o.sd.dump_ast(o.tree()))
    print('%s%s' % (' '*4*indent,o.format()))

<< define s >>
print('Input...\n%s\n' % (s.rstrip()))

m = leoInspect.module(s=s)

# print(show(m,0))

print('\nStatements...\n')
for o in m.statements():
    print(o.format())
    
if 0:
    print('\nAssignments...\n')
    for o in m.assignments():
        print(o.format())

print('\nAssignments to a...\n')
for o in m.assignments_to('a'):
    print(o.format())
    
print('\nAssignments using d...\n')
for o in m.assignments_using('d'):
    print(o.format())
    
print('\nCalls to f...\n')
for o in m.calls_to('f'):
    print(o.format())

if 0:
    for f in m.functions():
        show(f,0)
        for z in f.statements():
            show(z,1)
    
    for cls in m.classes():
        show(cls,0)
        for d in cls.defs():
            show(d,1)
            for z in d.statements():
                show(z,1)
.. @+node:ekr.20161117164321.44: *7* << define s >>
s = '''
x.y = b(arg1,arg2=5,*args,**args).c[1:2:3].d
a=b+c
p,d,q[5]=f(a=1,b=2,*args,**keys)
'''


# def outer_function(a,b=99,c=88,*args,**keys):
    # print('hello')
    
# class myClass:
    # def method():
        # pass
    
.. @+node:ekr.20161117164321.45: *5* leoInspect speed tests
.. @+node:ekr.20161117164321.46: *6* @test speed of leoInspect.module (all core files)
import leo.core.leoInspect as leoInspect
import time

t1 = time.time()

sd = leoInspect.SemanticData()
count = 0
for fn in leoInspect.LeoCoreFiles().files:
    # print(fn)
    m = leoInspect.module(fn,sd=sd)
    count += 1

t2 = time.time()

print('file: %s time: %2.2f sec' % (count,t2-t1))

if 0:
    sd.print_stats()
if 1:
    sd.print_times()
.. @+node:ekr.20161117164321.47: *6* @test speed of AstTraverser (all Leo core files)
import leo.core.leoGlobals as g
import leo.core.leoInspect as leoInspect
import ast
import time

read_time,parse_time,traverse_time = 0.0,0.0,0.0
t_start = time.time()
count = 0
for fn in leoInspect.LeoCoreFiles().files:
    count += 1
    t2 = time.time()
    s = open(fn,'r').read()
    t3 = time.time()
    tree = ast.parse(s,filename=fn,mode='exec')
    t4 = time.time()
    leoInspect.AstTraverser(fn).visit(tree)
    t5 = time.time()

    read_time += t3-t2
    parse_time += t4-t3
    traverse_time += t5-t4
t_end = time.time()
total_time = t_end-t_start
if 1:
    print('files:     %s' % (count))
    print('read:      %2.3f sec.' % (read_time))
    print('ast.parse: %2.3f sec.' % (parse_time))
    print('traverse:  %2.3f sec.' % (traverse_time))
    print('total:     %2.3f sec.' % (total_time))

.. @+node:ekr.20161117164321.48: *5* @test compute all ivars in all classes
import imp
import time

import leo.core.leoInspect as li
imp.reload(li)
g_dump,g_format,g_kind = li.g_dump,li.g_format,li.g_kind

t1 = time.time()

print('starting pass 1...')

# Globals...
sd = li.SemanticData(controller=None)
g_d = {} # Keys are ivars, values are lists of classes.
n_files = 0

for fn in li.LeoCoreFiles().files:

    # Most of the time is spent creating the context objects.
    m = li.module(fn=fn,sd=sd)
    n_files += 1
    
    ### To do: create global assignments list.
    
    if 1: # Look for all targets. Takes about 0.15 sec.
        for class_ in m.classes():
            for def_ in class_.defs():
                for a in def_.assignments():
                    tree = a.tree()
                    kind = g_kind(tree)
                    # if g_kind(tree.value) == 'ListComp':
                        # print(a.format())
                        # print(g_dump(tree.value))
                    # print(a.format())
                    if kind == 'Assign':
                        for target in tree.targets:
                            # if False and g_kind(target) not in ('Attribute','Name','Tuple','Subscript'):
                                # print('assn target: %s' % g_format(target))
                                # print('assn target: %s' % g_dump(target))
                            name = g_format(target)
                            if name.startswith('self.'):
                                name = name[5:]
                            i = name.find('[')
                            if i > -1:
                                name = name[:i]
                            aList = g_d.get(name,[])
                            if class_.name() not in aList:
                                aList.append(class_.name())
                                g_d[name] = aList
                    else:
                        assert kind == 'AugAssign',kind
                        name = g_format(tree.target)
                        aList = g_d.get(name,[])
                        if class_.name() not in aList:
                            aList.append(class_.name())
                            g_d[name] = aList

    # Takes about 0.1 sec. cumulative.
    if 0: # Look for all ivars.
        for class_ in m.classes():
            for def_ in class_.defs():
                if def_.name() == '__init__':
                    for a in def_.assignments_to('self'):
                        # Not all targets are ivars.
                        for target in a.tree().targets:
                            name = g_format(target)
                            if name.startswith('self.'):
                                name = name[5:]
                                aList = g_d.get(name,[])
                                if class_.name() not in aList:
                                    aList.append(class_.name())
                                    g_d[name] = aList
                       

t2 = time.time()

print('files: %s time: %2.2f sec, total ivars: %s' % (
    n_files,t2-t1,len(list(g_d.keys()))))

if 0:
    ambiguous,total = 0,0
    for key in sorted(g_d.keys()):
        aList = sorted(g_d.get(key))
        if 1 and len(aList) > 1:
            w = 30 # Width of left column
            if total == 0: print('Global ivars dict...')
            if len(key) + 3 > w: key = key[:w-3]+'...'
            aList2 = aList[:3]
            if len(aList2) < len(aList): aList2.append('...')
            print('%30s %2s %s' % (key,len(aList),aList2))
        total += 1
        if len(aList) > 1: ambiguous += 1
    print('total ivars: %s ambiguous: %s' % (total,ambiguous))

if 0:
    sd.print_stats()
.. @+node:ekr.20161117164321.49: *5* @test find all ctors
g.cls()

import imp
import time

import leo.core.leoInspect as li
imp.reload(li)
g_dump,g_format,g_kind = li.g_dump,li.g_format,li.g_kind

t1 = time.time()

# Globals...
sd = li.SemanticData(controller=None)
d = sd.modules_dict

# Pass 1: Load all modules.
print('starting pass 1...')
for fn in li.LeoCoreFiles().files:
    li.module(fn=fn,sd=sd)

t2 = time.time()
print('pass 1: %2.3f sec files: %s' % (t2-t1,len(list(d.keys()))))
    
# Pass 2: compute all class names.
classes = set()
for fn in sorted(d):
    m = d.get(fn)
    for class_ in m.classes():
        classes.add(class_.name())
        
t3 = time.time()
print('pass 2: %2.3f sec' % (t3-t2))

classes = sorted(list(classes))

if 0:
    for z in classes:
        print(z)
        
# Pass 3: Find all calls to ctors.
ctors = set()
ctors_assns = []
for fn in sorted(d):
    m = d.get(fn)
    for class_ in m.classes():
        for def_ in class_.defs():
            for a in def_.assignments():
                rhs = a.tree().value
                if m.tree_kind(rhs) == 'Call':
                    s = li.g_find_function_call(rhs.func)
                    if s in classes:
                        ctors.add(s)
                        ctors_assns.append(a.format())
 
t4 = time.time()                            

if 0:
    for z in sorted(list(ctors)):
        print(z)
if 0:
    for s in ctors_assns:
        aList = s.split('=')
        print('%30s = %s' % (aList[0],'='.join(aList[1:])[:80]))

print('pass 3: %2.3f sec ctors assigns: %s' % (t4-t3,len(ctors_assns)))

if 1:
    sd.print_stats()
if 1:
    sd.print_times()
.. @+node:ekr.20161117164321.50: *5* @test pickling
# g.cls()

import imp
import pickle
import time

import leo.core.leoInspect as li
imp.reload(li)
    
# Works, because TestPickleClass is a top-level class.
o = li.TestPickleClass()

try:
    s = pickle.dumps(o)
    print(len(s),o)
except pickle.PicklingError:
    print('can not pickle: %s' % repr(o))
.. @+node:ekr.20161117164321.51: *5* @test cache contexts
# g.cls()

import imp
import pickle
import time

import leo.core.leoInspect as li
imp.reload(li)
g_dump,g_format,g_kind = li.g_dump,li.g_format,li.g_kind

# Pass 1: Load all modules.
t1 = time.time()
sd = li.SemanticData(controller=None)
files = li.LeoCoreFiles().files

print('starting pass 1...')
for fn in files:
    li.module(fn=fn,sd=sd)

t2 = time.time()
print('pass 1: %2.3f sec files: %s' % (t2-t1,len(list(sd.modules_dict.keys()))))

# Pass 2: pickle all modules.
for fn in sorted(sd.modules_dict.keys()):
    m = sd.modules_dict.get(fn)
    try:
        s = pickle.dumps(m)
        print('%6s %s' % (len(s),fn))
    except pickle.PicklingError:
        print('can not pickle: %s' % repr(m))
        
t3 = time.time()
print('pass 2: %2.3f sec' % (t3-t2))
.. @+node:ekr.20161117164321.52: *5* @test AstFormatter
g.cls()

import imp

import leo.core.leoInspect as li
imp.reload(li)
g_format = li.g_format

# Pass 1: Load all modules.
sd = li.SemanticData(controller=None)
files = li.LeoCoreFiles().files # [0:2]

s = '''
def spam():
    """This is a docstring"""
    a = 2
    try:
        pass
    except Exception as message: ###
        pass
    if f(1):
        g('a')
    else:
        g(2)
    while 1 < 2:
        pass
    raise AttributeError ###
    return 2
'''

if 0: # String
    li.module(s=s,sd=sd)
else:
    print('starting pass 1...')
    for fn in files:
        li.module(fn=fn,sd=sd)
    
# Pass 2: format the module.
for fn in sorted(sd.modules_dict.keys()):
    m = sd.modules_dict.get(fn)
    # print(m.format())
    m.format() # Run for warnings.
.. @+node:ekr.20161117164321.53: *5* @test g_files_in_dir
import imp
import leo.core.leoInspect as li
imp.reload(li)

g.cls()

aList = li.g_files_in_dir(r'C:\Python26\Lib\lib2to3',
    extList = ['.py'],
    excludeDirs= ['tests'])

for z in aList:
    print(z)

print('files: %s' % (len(z)))
.. @+node:ekr.20161117164321.54: *5* @test print-chains
import ast
import imp
import time
import leo.core.leoInspect as li
imp.reload(li)

t1 = time.time()

g.cls()

last = c.rootPosition()
while last.hasNext():
    last = last.next()
    
parent = last.insertAfter()
parent.h = 'Chains: %s' % time.strftime('%Y/%m/%d/%H:%M:%S',time.localtime())
parent.b = '@killcolor'

count, total_chains,unusual_chains = 0,0,0
for fn in li.LeoCoreFiles().files: # [:2]:
    # print()
    # print(fn)
    s = open(fn,'r').read()
    tree = ast.parse(s,filename=fn,mode='exec')
    cp = li.ChainPrinter(fn)
    cp.visit(tree)
    p2 = parent.insertAsLastChild()
    p2.h = g.shortFileName(fn)
    n1,n2 = cp.showChains(p2)
    total_chains += n1
    unusual_chains += n2
    # print('chains: %s' % (n))
    count += 1
    
c.redraw(parent)

t2 = time.time()

print('files: %s total chains: %s unusual_chains: %s time: %2.2f sec' % (
    count,total_chains,unusual_chains,t2-t1))

# if 0:
    # sd.print_stats()
# if 0:
    # sd.print_times()
.. @+node:ekr.20161117164321.55: *5* @test print-calls
import ast
import imp
import time
import leo.core.leoInspect as li
imp.reload(li)

g.cls()

t1 = time.time()

last = c.rootPosition()
while last.hasNext():
    last = last.next()
    
parent = last.insertAfter()
parent.h = 'Calls: %s' % time.strftime('%Y/%m/%d/%H:%M:%S',time.localtime())
parent.b = '@killcolor'

g_d = {}
count=0
for fn in li.LeoCoreFiles().files: # [:3]:
    # print()
    # print(fn)
    s = open(fn,'r').read()
    tree = ast.parse(s,filename=fn,mode='exec')
    cp = li.CallPrinter(fn)
    cp.visit(tree)
    p2 = parent.insertAsLastChild()
    p2.h = g.shortFileName(fn)
    cp.showCalls(p2)
    for key in cp.d.keys():
        aList = g_d.get(key,[])
        aList.extend(cp.d.get(key))
        g_d[key] = sorted(list(set(aList)))
    count += 1
    
p2 = parent.insertAsLastChild()
p2.h = 'global calls'
cp.showCalls(p2,d=g_d)
c.selectPosition(parent)
c.redraw()

t2 = time.time()

print('files: %s time: %2.2f sec' % (count,t2-t1))
.. @+node:ekr.20161117164321.56: *5* @test print-returns
import ast
import imp
import time
import leo.core.leoInspect as li
imp.reload(li)

g.cls()

t1 = time.time()

last = c.rootPosition()
while last.hasNext():
    last = last.next()
    
project_name,verbose = 'leo',False # False: only print defs with more than one return.
files = li.g_get_files_by_project_name(project_name)

parent = last.insertAfter()
parent.h = 'Returns: %s verbose=%s %s' % (
    project_name, verbose,
    time.strftime('%Y/%m/%d/%H:%M:%S',time.localtime()))
parent.b = '@killcolor'

g_d = {}
count=0
for fn in files:
    # print(fn)
    s = open(fn,'r').read()
    try:
        tree = ast.parse(s,filename=fn,mode='exec')
    except SyntaxError:
        print('Syntax error in %s' % (fn))
        continue
    rp = li.ReturnPrinter(fn)
    rp.visit(tree)
    p2 = parent.insertAsLastChild()
    p2.h = g.shortFileName(fn)
    p2.b = rp.showReturns(verbose=verbose)
    for key in rp.d.keys():
        aList = g_d.get(key,[])
        aList2 = rp.d.get(key)
        if aList2:
            aList.extend(aList2)
            g_d[key] = aList
    count += 1
    
if 0:
    p2 = parent.insertAsLastChild()
    p2.h = 'global returns'
    rp.showReturns(p2,d=g_d)

c.redraw(parent)

t2 = time.time()

print('files: %s time: %2.2f sec' % (
    count,t2-t1))
.. @+node:ekr.20161117164321.57: *5* @test global names
# import ast
import imp
import time
import leo.core.leoInspect as li
imp.reload(li)

g.cls()

project_name,verbose = 'leo',False
files = li.g_get_files_by_project_name(project_name)
result = []

def put(s):
    result.append(s)
    # print(s)
    
t1 = time.time()

# Globals...
sd = li.SemanticData(controller=None)
m_d = sd.modules_dict

# Pass 1: Load all modules.
print('starting pass 1...')
for fn in files: ### [:2]:
    li.module(fn=fn,sd=sd)

t2 = time.time()
put('pass 1: %2.3f sec files: %s' % (t2-t1,len(list(m_d.keys()))))

# Pass 2: Update g_d.
g_d = {} # Keys are names, values are sets of Context names.
contexts = 0
for fn in sorted(m_d):
    m = m_d.get(fn)
    for cx in m.contexts(include_temp=True):
        # put(' '*len(cx.parent_contexts()),cx)
        contexts += 1
        d = cx.st.d # Keys are names, values are symbol table entries.
        for key in d.keys():
            e = d.get(key)
            name = e.name
            aSet = g_d.get(name,set())
            aSet.add(cx)
            g_d[name] = aSet
            
names = sorted(g_d.keys())
distribution = {} # Keys are lengths of context sets; values are number of ids with that length.
for key in names:
    aSet = g_d.get(key)
    context_list = sorted(list(set([repr(z) for z in aSet])))
    n = len(context_list)
    distribution[n] = distribution.get(n,0) + 1
    if verbose or n > 9:
        if n > 3:
            put('%20s %4s %s...' % (key,len(context_list),context_list[:3]))
        else:
            put('%20s %4s %s' % (key,len(context_list),context_list))
ids = len(names)
        
t3 = time.time()
put('pass 2: contexts: %s ids: %s %2.2f sec' % (contexts,ids,t3-t2))

if 1:
    put('\nDistribution of context lengths...')
    for key in sorted(distribution.keys()):
        put('%4s %s' % (key,distribution.get(key)))
    
if 1:
    last = c.rootPosition()
    while last.hasNext():
        last = last.next()
    parent = last.insertAfter()
    parent.h = 'Global names: %s verbose=%s %s' % (
        project_name, verbose,
        time.strftime('%Y/%m/%d/%H:%M:%S',time.localtime()))
    parent.b = '@killcolor\n\n%s' % '\n'.join(result)
    c.redraw(parent)

if 0:
    sd.print_stats()
if 0:
    sd.print_times()

.. @+node:ekr.20161117164321.58: *5* @test leoInspect.token_range (s)
import leo.core.leoInspect as inspect

# g.cls()

testing = g.unitTesting

def show(o,indent=0):
    pad = ' '*4*indent
    if not testing:
        # print('\n%s\n' % o.sd.dump_ast(o.tree()))
        print('%s%s' % (pad,o.format()))
        print('token range: %s' % (repr(o.token_range())))

<< define s >>
if not testing:
    print('Input...\n%s\n' % (s.rstrip()))

m = inspect.module(s=s)

if 0:
    show(m,0)
    
if 1:
    if not testing: print('\nAssignments to a...\n')
    for o in m.assignments_to('a'):
        # print(o.format())
        show(o)
if 0:
    if not testing: print('\nAssignments using d...\n')
    for o in m.assignments_using('d'):
        # print(o.format())
        show(o)
if 0:    
    if not testing: print('\nCalls to f...\n')
    for o in m.calls_to('f'):
        # print(o.format())
        show(o)
if 0:
    for s in m.statements():
        show(s)
    
    for f in m.functions():
        show(f,0)
        for z in f.statements():
            show(z,1)
    
    for cls in m.classes():
        show(cls,0)
        for d in cls.defs():
            show(d,1)
            for z in d.statements():
                show(z,1)
.. @+node:ekr.20161117164321.59: *6* << define s >>
s = '''
# x.y = b(arg1,arg2=5,*args,**args).c[1:2:3].d
a=b+c
# f(a=1,b=2,*args,**keys)
'''


# def outer_function(a,b=99,c=88,*args,**keys):
    # print('hello')
    
# class myClass:
    # def method():
        # pass
    
.. @+node:ekr.20161117164321.60: *4* @ignore leoViews.py
.. @+node:ekr.20161117164321.61: *5* @@@test vc.create_tree_structure (rewrite)
vc = c.viewController
views = g.findNodeAnywhere(c,'@views')
if views: views.deleteAllChildren()
    # Start with a pristine @views tree.
root_before  = g.findNodeInTree(c,p,'root_before')
    # Root before: used to generate @auto-view tree.
root_restore = g.findNodeInTree(c,p,'root_restore')
    # Create root_after from root_restore.
assert root_before,root_restore
root_after  = g.findNodeInTree(c,p,'root_after')
if root_after: root_after.doDelete()
    # Root after: the results of the previous test.
c.selectPosition(root_restore)
c.copyOutline()
c.selectPosition(root_before)
c.pasteOutline()
# The roots must look like @auto nodes.
root_after = c.p
assert root_after.h == 'root_restore',root_after.h
root_before.h = '@auto root_before'
root_after.h = '@auto root_after' 
try:
    vc.update_before_write_at_auto_file(root_before)
    at_organizers = vc.has_at_organizers_node(root_before)
    assert at_organizers
    # Called by vc.create_organizer_nodes(organizers,root_after):
    root = root_after
    vc.create_organizer_data(at_organizers,root)
    vc.create_actual_organizer_nodes()
    vc.create_tree_structure(root)
    # The body of demote_organized_nodes:
    for od in vc.all_ods:
        # Called by vc.update_helper.
        od_list = vc.find_all_organizer_nodes(od)
        assert od in od_list,od_list
    d = {
        'organizer node': ['intermediate node','inner org1','inner org2'],
        'intermediate node': ['inner org1','inner org2'],
    }
    for od in vc.organizer_data_list:
        aList = d.get(od.h,[])
        aList2 = [z.h for z in od.descendants or []]
        assert sorted(aList) == sorted(aList2),(aList,aList2)
finally:
    # Make sure the roots are *not* @auto nodes.
    vc.temp_node.doDelete()
    root_before.h = 'root_before'
    root_after.h = 'root_after'
    c.redraw()
.. @+node:ekr.20161117164321.62: *6* root_restore
@others
.. @+node:ekr.20161117164321.63: *7* aClass
class aClass:
    @others
.. @+node:ekr.20161117164321.64: *8* before
def spam_before(self):
    pass
.. @+node:ekr.20161117164321.65: *8* child11
def spam11(self):
    pass
.. @+node:ekr.20161117164321.66: *8* child12
def spam12(self):
    pass
.. @+node:ekr.20161117164321.67: *8* middle
def middle_spam():
    pass
.. @+node:ekr.20161117164321.68: *8* child21
def spam21(self):
    pass
.. @+node:ekr.20161117164321.69: *8* child22
def spam22(self):
    pass
.. @+node:ekr.20161117164321.70: *8* after
def spam_after(self):
    pass
.. @+node:ekr.20161117164321.71: *8* last1
def spam_last1():
    pass
.. @+node:ekr.20161117164321.72: *7* top1
def spam_top1():
    pass
.. @+node:ekr.20161117164321.73: *7* top2
def spam_top2():
    pass
.. @+node:ekr.20161117164321.74: *6* root_before
@others
.. @+node:ekr.20161117164321.75: *7* aClass
class aClass:
    @others
.. @+node:ekr.20161117164321.76: *8* before
def spam_before(self):
    pass
.. @+node:ekr.20161117164321.77: *8* organizer node
.. @+node:ekr.20161117164321.78: *9* intermediate node
.. @+node:ekr.20161117164321.79: *10* inner org1
.. @+node:ekr.20161117164321.80: *11* child11
def spam11(self):
    pass
.. @+node:ekr.20161117164321.81: *11* child12
def spam12(self):
    pass
.. @+node:ekr.20161117164321.82: *10* middle
def middle_spam():
    pass
.. @+node:ekr.20161117164321.83: *10* inner org2
.. @+node:ekr.20161117164321.84: *11* child21
def spam21(self):
    pass
.. @+node:ekr.20161117164321.85: *11* child22
def spam22(self):
    pass
.. @+node:ekr.20161117164321.86: *8* after
def spam_after(self):
    pass
.. @+node:ekr.20161117164321.87: *8* last organizer
.. @+node:ekr.20161117164321.88: *9* last1
def spam_last1():
    pass
.. @+node:ekr.20161117164321.89: *7* top-level
.. @+node:ekr.20161117164321.90: *8* top1
def spam_top1():
    pass
.. @+node:ekr.20161117164321.91: *8* top2
def spam_top2():
    pass
.. @+node:ekr.20161117164321.92: *6* root_after
@others
.. @+node:ekr.20161117164321.93: *7* aClass
class aClass:
    @others
.. @+node:ekr.20161117164321.94: *8* before
def spam_before(self):
    pass
.. @+node:ekr.20161117164321.95: *8* child11
def spam11(self):
    pass
.. @+node:ekr.20161117164321.96: *8* child12
def spam12(self):
    pass
.. @+node:ekr.20161117164321.97: *8* middle
def middle_spam():
    pass
.. @+node:ekr.20161117164321.98: *8* child21
def spam21(self):
    pass
.. @+node:ekr.20161117164321.99: *8* child22
def spam22(self):
    pass
.. @+node:ekr.20161117164321.100: *8* after
def spam_after(self):
    pass
.. @+node:ekr.20161117164321.101: *8* last1
def spam_last1():
    pass
.. @+node:ekr.20161117164321.102: *7* top1
def spam_top1():
    pass
.. @+node:ekr.20161117164321.103: *7* top2
def spam_top2():
    pass
.. @+node:ekr.20161117164321.104: *5* @test atFile.new_auto
# To remind ourselves of the status of new_import.
import leo.core.leoAtFile as atFile
print('new_import: %s' % atFile.new_auto)
.. @+node:ekr.20161117164321.105: *5* @test p.sort_key
aList = [p.copy() for p in c.all_positions()]
aList2 = sorted(reversed(aList),key=p.sort_key)
i = 0
for p in aList2:
    p2 = aList[i]
    i += 1
    assert p == p2,'\n%s:%s\n%s:%s' % (
        p.sort_key(p),p.h,p2.sort_key(p2),p2.h)
.. @+node:ekr.20161117164321.106: *5* @test vc.clean_nodes (to do)
vc = c.viewController
views = g.findNodeAnywhere(c,'@views')
assert views
views.deleteAllChildren()
views.b = None ####
root1 = g.findNodeInTree(c,p,'root1')
assert root1
try:
    root1.h = '@auto root1'
finally:
    root1.h = 'root1'
.. @+node:ekr.20161117164321.107: *6* root1
.. @+node:ekr.20161117164321.108: *6* root2
.. @+node:ekr.20161117164321.109: *5* @test vc.create_clone_links
vc = c.viewController
clones = g.findNodeInTree(c,p,'@clones')
root = g.findNodeInTree(c,p,'root')
node1 = g.findNodeInTree(c,p,'node1')
assert clones and root and node1
root.deleteAllChildren()
new_node1 = root.insertAsLastChild()
new_node1.h = 'node1'
clones.b = 'gnx: %s\nunl: %s\n' % (node1.v.gnx,'node1')
try:
    ok = vc.create_clone_links(clones,root)
    assert ok
    # Important: p._relinkAsCloneOf leaves new_node1 unchanged,
    # but new_node1 should not be used.
finally:
    c.redraw()
.. @+node:ekr.20161117164321.110: *6* @clones
gnx: ekr.20140211085929.5552
unl: node1
.. @+node:ekr.20161117164321.113: *6* node1
.. @+node:ekr.20161117164321.112: *6* root
.. @+node:ekr.20161117164321.113: *7* node1
.. @+node:ekr.20161117164321.114: *5* @test vc.create_organizer_node
vc = c.viewController
views = g.findNodeAnywhere(c,'@views')
if views: views.deleteAllChildren()
root_restore = g.findNodeInTree(c,p,'root_restore')
assert root_restore
root_before = g.findNodeInTree(c,p,'root_before')
assert root_before
root_after = g.findNodeInTree(c,p,'root_after')
if root_after: root_after.doDelete()
    # Root after: the results of the previous test.
c.selectPosition(root_restore)
c.copyOutline()
c.selectPosition(root_before)
c.pasteOutline()
root_after = c.p
try:
    # The roots must look like @auto nodes.
    root_before.h = '@auto root_before'
    root_after.h = '@auto root_after'
    vc.init() # Required.
    vc.update_before_write_at_auto_file(root_before)
    organizers = vc.has_at_organizers_node(root_before)
    assert organizers
    vc.create_organizer_nodes(organizers,root_after)
    ok = vc.compare_test_trees(root_before,root_after)
    assert ok,'\n\nexpected...\n%s\ngot...\n%s' % (
        vc.trial_write(root_before),vc.trial_write(root_after))
finally:
     # Make sure the roots are *not* @auto nodes.
    root_before.h = 'root_before'
    root_after.h = 'root_after'
    # Replace root_after by a copy of root_restore.
    # This ensures that the unit test can be run more than once.
    if 0:
        root_after.doDelete()
        c.selectPosition(root_restore)
        c.copyOutline()
        c.selectPosition(root_before)
        c.pasteOutline()
        assert c.p.h == 'root_restore'
        c.p.h = 'root_after'
        c.redraw()
.. @+node:ekr.20161117164321.115: *6* root_restore
@others
.. @+node:ekr.20161117164321.116: *7* aClass
class aClass:
    @others
.. @+node:ekr.20161117164321.117: *8* before
def before():
    pass
.. @+node:ekr.20161117164321.118: *8* child1
def child1():
    pass
.. @+node:ekr.20161117164321.119: *8* extra2
def extra2():
    pass
.. @+node:ekr.20161117164321.120: *8* child2
def child2():
    pass
.. @+node:ekr.20161117164321.121: *8* after
def after():
    pass
.. @+node:ekr.20161117164321.122: *6* root_before
@others
.. @+node:ekr.20161117164321.123: *7* aClass
class aClass:
    @others
.. @+node:ekr.20161117164321.124: *8* before
def before():
    pass
.. @+node:ekr.20161117164321.125: *8* organizer node
.. @+node:ekr.20161117164321.126: *9* child1
def child1():
    pass
.. @+node:ekr.20161117164321.127: *9* extra2
def extra2():
    pass
.. @+node:ekr.20161117164321.128: *9* child2
def child2():
    pass
.. @+node:ekr.20161117164321.129: *8* after
def after():
    pass
.. @+node:ekr.20161117164321.130: *6* root_after
@others
.. @+node:ekr.20161117164321.131: *7* aClass
class aClass:
    @others
.. @+node:ekr.20161117164321.132: *8* before
def before():
    pass
.. @+node:ekr.20161117164321.133: *8* organizer node
.. @+node:ekr.20161117164321.134: *9* child1
def child1():
    pass
.. @+node:ekr.20161117164321.135: *9* extra2
def extra2():
    pass
.. @+node:ekr.20161117164321.136: *9* child2
def child2():
    pass
.. @+node:ekr.20161117164321.137: *8* after
def after():
    pass
.. @+node:ekr.20161117164321.138: *5* @test vc.drop_all_organizers_in_unl
vc = c.viewController
organizer_unls = [
    'a-->O1',
    'a-->O1-->O2',
    'a-->O3',
    'a-->O3-->b-->O4',
    'z',
]
table = (
    ('z-->x','x'), # Test of dropping a leading -->.
    ('a-->O1-->unl1','a-->unl1'),
    ('a-->O1-->O2-->unl2','a-->unl2'),
    ('a-->O3-->unl3','a-->unl3'),
    ('a-->O3-->b-->O4-->unl4','a-->b-->unl4'),
)
for unl,expected in table:
    got = vc.drop_all_organizers_in_unl(organizer_unls,unl)
    assert expected == got,'\nunl:      %s\nexpected: %s\ngot:      %s' % (unl,expected,got)
.. @+node:ekr.20161117164321.139: *5* @test vc.find_absolute_unl_node
vc = c.viewController
root = c.rootPosition().insertAfter()
root.h = 'root'
child1 = root.insertAsLastChild()
child1.h = 'child1'
child2 = child1.insertAfter()
child2.h = 'child2'
child11 = child1.insertAsLastChild()
child11.h = 'child11'
try:
    for unl in ('root','root-->child1','root-->child2','root-->child1-->child11'):
        p = vc.find_absolute_unl_node(unl)
        parts = unl.split('-->')
        assert p,unl
        assert p.h == parts[-1],p.h
finally:
    root.doDelete()
    c.selectPosition(p)
    c.redraw()
.. @+node:ekr.20161117164321.140: *5* @test vc.find_at_views_node
vc = c.viewController
tag = '@views'
views = g.findNodeAnywhere(c,tag)
assert views
views2 = vc.has_at_views_node()
assert views == views2,(views,views2)
.. @+node:ekr.20161117164321.141: *5* @test vc.find_position_for_relative_unl
vc = c.viewController
parent = p.copy()
node1 = p.firstChild()
node2 = node1.next()
assert node1 and node2
child11 = node1.firstChild()
child12 = child11.next()
assert child11 and child12
child21 = node2.firstChild()
child22 = child21.next()
assert child21 and child22
table = (
    # ('node1',node1),
    ('',parent), # This special case is important.
    ('node1-->child11',child11),
    ('node1-->child12',child12),
    ('node2',node2),
    ('node2-->child21',child21),
    ('node2-->child22',child22),
    ('node3',None),
    ('node1-->childx',None),
    ('node3-->childx',None),
)
for unl,expected in table:
    got = vc.find_position_for_relative_unl(parent,unl)
    assert got == expected,'unl: %s expected: %s got: %s' % (
        unl,expected and expected.h,got and got.h)
.. @+node:ekr.20161117164321.142: *6* node1
.. @+node:ekr.20161117164321.143: *7* child11
.. @+node:ekr.20161117164321.144: *7* child12
.. @+node:ekr.20161117164321.145: *6* node2
.. @+node:ekr.20161117164321.146: *7* child21
.. @+node:ekr.20161117164321.147: *7* child22
.. @+node:ekr.20161117164321.148: *5* @test vc.find_representative_node
vc = c.viewController
root = g.findNodeInTree(c,p,'root')
assert root
root.h = '@auto root'
try:
    clone = root.next()
    assert clone and clone.h == 'clone'
    inner_clone = root.firstChild()
    assert inner_clone
    assert clone.v == inner_clone.v
    rep = vc.find_representative_node(root,inner_clone)
    # Careful: cloning this test can cause problems.
    oops = '\n  rep: %s\nparent:%s\nclone: %s\nparent:%s\ninner: %s\nparent: %s' % (
        rep,rep.parent(),clone,clone.parent(),inner_clone,inner_clone.parent())
    if True: ### p.isCloned():
        assert rep.v == clone.v and rep.parent().v == clone.parent().v,oops
    else:
        assert rep == clone
finally:
    root.h = 'root' # root must not be an @auto node.
    c.redraw()
.. @+node:ekr.20161117164321.149: *6* root
.. @+node:ekr.20161117164321.151: *7* clone
.. @+node:ekr.20161117164321.151: *6* clone
.. @+node:ekr.20161117164321.152: *5* @test vc.find_views/clones/organizers_node
# Also a test of find_at_views_node, find_at_organizers_node and find_at_clones_node.
vc = c.viewController
root = g.findNodeInTree(c,p,'root')
assert root
views = g.findNodeAnywhere(c,'@views')
assert views,'1'
if views:
    views.deleteAllChildren()
try:
    root.h = '@auto root' # root must look like an @auto node.
    views = vc.find_at_views_node()
    assert views,'2'
    views2 = vc.find_at_views_node()
    assert views2 == views
    assert vc.find_at_clones_node(root)
    assert vc.find_at_organizers_node(root)
finally:
    root.h = 'root' # Make sure root is *not* an @auto node.
    # views.deleteAllChildren()
    c.selectPosition(p)
    c.redraw()
.. @+node:ekr.20161117164321.153: *6* root
.. @+node:ekr.20161117164321.154: *7* aClass
class aClass:
    @others
.. @+node:ekr.20161117164321.158: *8* clone
.. @+node:ekr.20161117164321.156: *7* organizer node
.. @+node:ekr.20161117164321.157: *8* child2
def spam():
    pass
.. @+node:ekr.20161117164321.158: *6* clone
.. @+node:ekr.20161117164321.159: *5* @test vc.has_*_node
# Test vc.has_at_auto_view_node, vc.has_at_clones_node and vc.has_at_organizers_node.
vc = c.viewController
views = g.findNodeAnywhere(c,'@views')
if views:
    assert vc.has_at_views_node()
    views.deleteAllChildren()
else:
    assert not vc.has_at_views_node()
    views = vc.find_at_views_node()
assert views
root = g.findNodeInTree(c,p,'root')
assert root
# The representative of clone_test node must appear outside of root's tree.
clone_test = g.findNodeInTree(c,p,'clone-test')
assert clone_test
assert clone_test.v == root.next().v,(clone_test.v,root.next().v)
try:
    root.h = '@auto root' # root must look like an @auto node.
    vc.update_before_write_at_auto_file(root)
    auto_view = g.findNodeInTree(c,views,'@auto-view:root')
    assert auto_view
    auto_view2 = vc.has_at_auto_view_node(root)
    assert auto_view2
    assert auto_view2 == auto_view,(auto_view,auto_view2)
    clones = g.findNodeInTree(c,auto_view,'@clones')
    assert clones
    clones2 = vc.has_at_clones_node(root)
    assert clones2
    assert clones2 == clones
    organizers = g.findNodeInTree(c,auto_view,'@organizers')
    assert organizers
    organizers2 = vc.has_at_organizers_node(root)
    assert organizers2
    assert organizers2 == organizers
finally:
    root.h = 'root' # Make sure root is *not* an @auto node.
    c.redraw()
.. @+node:ekr.20161117164321.160: *6* root
@others
.. @+node:ekr.20161117164321.161: *7* aClass
class aClass:
    @others
.. @+node:ekr.20161117164321.165: *8* clone-test
def clone_test():
    pass
.. @+node:ekr.20161117164321.163: *8* organizer node
.. @+node:ekr.20161117164321.164: *9* child2
def spam():
    pass
.. @+node:ekr.20161117164321.165: *6* clone-test
def clone_test():
    pass
.. @+node:ekr.20161117164321.166: *5* @test vc.is_at_auto_node
vc = c.viewController
p.deleteAllChildren()
auto = p.insertAsLastChild()
auto.h = '@auto test.py'
auto2 = p.insertAsLastChild()
auto2.h = '@auto-rst test2.py'
try:
    assert vc.is_at_auto_node(auto)
    assert not vc.is_at_auto_node(auto2)
finally:
    # This is required.
    p.deleteAllChildren()
    c.redraw()
.. @+node:ekr.20161117164321.167: *5* @test vc.is_organizer_node
vc = c.viewController
redraw_flag = False
for child in p.children():
    # Add a child so the test doesn't depend on that.
    if not child.hasChildren():
        child2 = child.insertAsLastChild()
        child2.h = 'child'
        redraw_flag = True
        
    expected = child.h.strip().endswith('True')
    got = vc.is_organizer_node(child,child)
    assert expected == got,'expected: %s in: %s body...\n%s' % (
        expected,child.h,child.b)
if redraw_flag:
    c.redraw()
.. @+node:ekr.20161117164321.168: *6* test python True
@language python

# An organizer node

# Another line.
.. @+node:ekr.20161117164321.169: *7* child
.. @+node:ekr.20161117164321.170: *6* test python 2
@language python

def spam():
    pass
.. @+node:ekr.20161117164321.171: *7* child
.. @+node:ekr.20161117164321.172: *6* test html True
@language html

<!-- comment -->

<!-- comment
continued comment
-->

.. @+node:ekr.20161117164321.173: *7* child
.. @+node:ekr.20161117164321.174: *6* test html 2
@language html

<!-- comment -->

<p> oops </p>

<!-- comment
continued comment
-->

.. @+node:ekr.20161117164321.175: *7* child
.. @+node:ekr.20161117164321.176: *5* @test vc.unl
vc = c.viewController
unl = vc.unl(p)
assert unl.endswith('-->'+p.h),repr(unl)
.. @+node:ekr.20161117164321.177: *5* @test vc.update_before_write_at_auto_file
vc = c.viewController
root = g.findNodeInTree(c,p,'root')
assert root
views = g.findNodeAnywhere(c,'@views')
if views:
    views.deleteAllChildren()
try:
    root.h = '@auto root' # root must look like an @auto node.
    vc.update_before_write_at_auto_file(root)
    views = g.findNodeAnywhere(c,'@views')
    assert views
    clones = g.findNodeInTree(c,views,'@clones')
    assert clones
    assert clones.b.endswith('aClass-->clone\n'),repr(clones.b)
    organizer = g.findNodeInTree(c,views,'@organizer: organizer node')
    assert organizer
    s1 = 'unl: organizer node-->child1'
    s2 = 'unl: organizer node-->child2'
    assert organizer.b == '\n'.join([s1,s2]),organizer.b
finally:
    root.h = 'root' # Make sure root is *not* an @auto node.
    if False and views:
        views.deleteAllChildren()
    c.redraw()
.. @+node:ekr.20161117164321.178: *6* root
.. @+node:ekr.20161117164321.179: *7* aClass
class aClass:
    @others
.. @+node:ekr.20161117164321.184: *8* clone
.. @+node:ekr.20161117164321.181: *7* organizer node
.. @+node:ekr.20161117164321.182: *8* child1
.. @+node:ekr.20161117164321.183: *8* child2
def spam():
    pass
.. @+node:ekr.20161117164321.184: *6* clone
.. @+node:ekr.20161117164321.185: *5* @test view-pack & view_unpack
vc = c.viewController
views = g.findNodeAnywhere(c,'@views')
assert views
views.deleteAllChildren()
view = g.findNodeInTree(c,p,'@view test')
assert view
assert c.positionExists(view)
try:
    c.selectPosition(view)
    v_b = view.b
    vc.pack()
    assert c.p.v == view.v
    vc.unpack()
    assert view.b == v_b,view.b
    assert view.lastChild().isCloned()
finally:
    # views.deleteAllChildren()
    c.undoer.clearUndoState()
    c.redraw()
.. @+node:ekr.20161117164321.189: *6* clone
clone body
.. @+node:ekr.20161117164321.187: *6* @view test
view body
.. @+node:ekr.20161117164321.188: *7* not a clone
not a clone text
.. @+node:ekr.20161117164321.189: *7* clone
clone body
.. @+node:ekr.20161117164321.190: *5* @test vc.create_organizer_node (nested organizers)
vc = c.viewController
views = g.findNodeAnywhere(c,'@views')
if views: views.deleteAllChildren()
    # Start with a pristine @views tree.
root_before  = g.findNodeInTree(c,p,'root_before')
    # Root before: used to generate @auto-view tree.
root_restore = g.findNodeInTree(c,p,'root_restore')
    # Create root_after from root_restore.
assert root_before,root_restore
root_after  = g.findNodeInTree(c,p,'root_after')
if root_after: root_after.doDelete()
    # Root after: the results of the previous test.
c.selectPosition(root_restore)
c.copyOutline()
c.selectPosition(root_before)
c.pasteOutline()
# The roots must look like @auto nodes.
root_after = c.p
assert root_after.h == 'root_restore',root_after.h
root_before.h = '@auto root_before'
root_after.h = '@auto root_after' 
try:
    vc.init() # Required.
    vc.update_before_write_at_auto_file(root_before)
    at_organizers = vc.has_at_organizers_node(root_before)
    assert at_organizers
    vc.create_organizer_nodes(at_organizers,root_after)
    ok = vc.compare_test_trees(root_before,root_after)
    assert ok,'\n\nexpected...\n%s\ngot...\n%s' % (
        vc.trial_write(root_before),vc.trial_write(root_after))
finally:
     # Make sure the roots are *not* @auto nodes.
    root_before.h = 'root_before'
    root_after.h = 'root_after'
    c.redraw()
.. @+node:ekr.20161117164321.191: *6* root_restore
@others
.. @+node:ekr.20161117164321.192: *7* aClass
class aClass:
    @others
.. @+node:ekr.20161117164321.193: *8* before
def before(self):
    pass
.. @+node:ekr.20161117164321.194: *8* child11
def child11(self):
    pass
.. @+node:ekr.20161117164321.195: *8* child12
def child12(self):
    pass
.. @+node:ekr.20161117164321.196: *8* middle
def middle():
    pass
.. @+node:ekr.20161117164321.197: *8* child21
def child21(self):
    pass
.. @+node:ekr.20161117164321.198: *8* child22
def child22(self):
    pass
.. @+node:ekr.20161117164321.199: *8* after
def after(self):
    pass
.. @+node:ekr.20161117164321.200: *8* last1
def last1():
    pass
.. @+node:ekr.20161117164321.201: *7* top1
def top1():
    pass
.. @+node:ekr.20161117164321.202: *7* top2
def top2():
    pass
.. @+node:ekr.20161117164321.203: *6* root_before
@others
.. @+node:ekr.20161117164321.204: *7* aClass
class aClass:
    @others
.. @+node:ekr.20161117164321.205: *8* before
def before(self):
    pass
.. @+node:ekr.20161117164321.206: *8* organizer node
.. @+node:ekr.20161117164321.207: *9* inner org1
.. @+node:ekr.20161117164321.208: *10* child11
def child11(self):
    pass
.. @+node:ekr.20161117164321.209: *10* child12
def child12(self):
    pass
.. @+node:ekr.20161117164321.210: *9* middle
def middle():
    pass
.. @+node:ekr.20161117164321.211: *9* inner org2
.. @+node:ekr.20161117164321.212: *10* child21
def child21(self):
    pass
.. @+node:ekr.20161117164321.213: *10* child22
def child22(self):
    pass
.. @+node:ekr.20161117164321.214: *8* after
def after(self):
    pass
.. @+node:ekr.20161117164321.215: *8* last organizer
.. @+node:ekr.20161117164321.216: *9* last1
def last1():
    pass
.. @+node:ekr.20161117164321.217: *7* top-level
.. @+node:ekr.20161117164321.218: *8* top1
def top1():
    pass
.. @+node:ekr.20161117164321.219: *8* top2
def top2():
    pass
.. @+node:ekr.20161117164321.220: *6* root_after
@others
.. @+node:ekr.20161117164321.221: *7* aClass
class aClass:
    @others
.. @+node:ekr.20161117164321.222: *8* before
def before(self):
    pass
.. @+node:ekr.20161117164321.223: *8* organizer node
.. @+node:ekr.20161117164321.224: *9* inner org1
.. @+node:ekr.20161117164321.225: *10* child11
def child11(self):
    pass
.. @+node:ekr.20161117164321.226: *10* child12
def child12(self):
    pass
.. @+node:ekr.20161117164321.227: *9* middle
def middle():
    pass
.. @+node:ekr.20161117164321.228: *9* inner org2
.. @+node:ekr.20161117164321.229: *10* child21
def child21(self):
    pass
.. @+node:ekr.20161117164321.230: *10* child22
def child22(self):
    pass
.. @+node:ekr.20161117164321.231: *8* after
def after(self):
    pass
.. @+node:ekr.20161117164321.232: *8* last organizer
.. @+node:ekr.20161117164321.233: *9* last1
def last1():
    pass
.. @+node:ekr.20161117164321.234: *5* @test vc.create_organizer_node (intermediate organizers)
vc = c.viewController
views = g.findNodeAnywhere(c,'@views')
if views: views.deleteAllChildren()
    # Start with a pristine @views tree.
root_before  = g.findNodeInTree(c,p,'root_before')
    # Root before: used to generate @auto-view tree.
root_restore = g.findNodeInTree(c,p,'root_restore')
    # Create root_after from root_restore.
assert root_before,root_restore
root_after  = g.findNodeInTree(c,p,'root_after')
if root_after: root_after.doDelete()
    # Root after: the results of the previous test.
c.selectPosition(root_restore)
c.copyOutline()
c.selectPosition(root_before)
c.pasteOutline()
# The roots must look like @auto nodes.
root_after = c.p
assert root_after.h == 'root_restore',root_after.h
root_before.h = '@auto root_before'
root_after.h = '@auto root_after' 
try:
    vc.init() # Required.
    vc.update_before_write_at_auto_file(root_before)
    at_organizers = vc.has_at_organizers_node(root_before)
    assert at_organizers
    vc.create_organizer_nodes(at_organizers,root_after)
    ok = vc.compare_test_trees(root_before,root_after)
    assert ok,'\n\nexpected...\n%s\ngot...\n%s' % (
        vc.trial_write(root_before),vc.trial_write(root_after))
finally:
     # Make sure the roots are *not* @auto nodes.
    root_before.h = 'root_before'
    root_after.h = 'root_after'
    c.redraw()
.. @+node:ekr.20161117164321.235: *6* root_restore
@others
.. @+node:ekr.20161117164321.236: *7* aClass
class aClass:
    @others
.. @+node:ekr.20161117164321.237: *8* before
def before(self):
    pass
.. @+node:ekr.20161117164321.238: *8* child11
def child11(self):
    pass
.. @+node:ekr.20161117164321.239: *8* child12
def child12(self):
    pass
.. @+node:ekr.20161117164321.240: *8* middle
def middle():
    pass
.. @+node:ekr.20161117164321.241: *8* child21
def child21(self):
    pass
.. @+node:ekr.20161117164321.242: *8* child22
def child22(self):
    pass
.. @+node:ekr.20161117164321.243: *8* after
def after(self):
    pass
.. @+node:ekr.20161117164321.244: *8* last1
def last1():
    pass
.. @+node:ekr.20161117164321.245: *7* top1
def top1():
    pass
.. @+node:ekr.20161117164321.246: *7* top2
def top2():
    pass
.. @+node:ekr.20161117164321.247: *6* root_before
@others
.. @+node:ekr.20161117164321.248: *7* aClass
class aClass:
    @others
.. @+node:ekr.20161117164321.249: *8* before
def before(self):
    pass
.. @+node:ekr.20161117164321.250: *8* organizer node
.. @+node:ekr.20161117164321.251: *9* inner org1
.. @+node:ekr.20161117164321.252: *10* child11
def child11(self):
    pass
.. @+node:ekr.20161117164321.253: *10* child12
def child12(self):
    pass
.. @+node:ekr.20161117164321.254: *9* middle
def middle():
    pass
.. @+node:ekr.20161117164321.255: *9* inner org2
.. @+node:ekr.20161117164321.256: *10* child21
def child21(self):
    pass
.. @+node:ekr.20161117164321.257: *10* child22
def child22(self):
    pass
.. @+node:ekr.20161117164321.258: *8* after
def after(self):
    pass
.. @+node:ekr.20161117164321.259: *8* last organizer
.. @+node:ekr.20161117164321.260: *9* last1
def last1():
    pass
.. @+node:ekr.20161117164321.261: *7* top-level
.. @+node:ekr.20161117164321.262: *8* top1
def top1():
    pass
.. @+node:ekr.20161117164321.263: *8* top2
def top2():
    pass
.. @+node:ekr.20161117164321.264: *6* root_after
@others
.. @+node:ekr.20161117164321.265: *7* aClass
class aClass:
    @others
.. @+node:ekr.20161117164321.266: *8* before
def before(self):
    pass
.. @+node:ekr.20161117164321.267: *8* organizer node
.. @+node:ekr.20161117164321.268: *9* inner org1
.. @+node:ekr.20161117164321.269: *10* child11
def child11(self):
    pass
.. @+node:ekr.20161117164321.270: *10* child12
def child12(self):
    pass
.. @+node:ekr.20161117164321.271: *9* middle
def middle():
    pass
.. @+node:ekr.20161117164321.272: *9* inner org2
.. @+node:ekr.20161117164321.273: *10* child21
def child21(self):
    pass
.. @+node:ekr.20161117164321.274: *10* child22
def child22(self):
    pass
.. @+node:ekr.20161117164321.275: *8* after
def after(self):
    pass
.. @+node:ekr.20161117164321.276: *8* last organizer
.. @+node:ekr.20161117164321.277: *9* last1
def last1():
    pass
.. @+node:ekr.20161117164321.278: *4* @ignore leoVim
.. @+node:ekr.20161117164321.279: *5* Unused
.. @+node:ekr.20161117164321.280: *6* @@test command regex
@language python
# http://docs.python.org/2/library/re.html
import re
# g.cls()
trace = False
n =     r'(?P<n>[0-9]*)'    # Optional digits
cmd =   r'(?P<cmd>[^0-9]+)' # Required: anything *except* digits.
n2 =    r'(?P<n2>[0-9]*)'   # Optional digits
cmd2 =  r'(?P<cmd2>[a-zA-Z]?)' # Optional letter.
n_c = n+cmd+n2+cmd2
tables = (
    (n_c,('35N','N','2d2','d2d','gg',)),
)
for pat,aList in tables:
    fields = re.findall('\(\?P<([a-z_A-Z0-9]+)>',pat)
    if trace: print('pattern: %s\n fields: %s' % (pat,','.join(fields)))
    for s in aList:
        if trace: print('  %s' % s)
        m = re.search(pat,s)
        for field in fields:
            try:
                val = m.group(field)
            except Exception:
                g.es_exception()
                val = None
            if trace: print('    %7s %s' % (field,val or 'None'))
.. @+node:ekr.20161117164321.281: *6* @@test motion regex
@language python

# http://docs.python.org/2/library/re.html
import re
# g.cls()

def escape(ch):
    return ch if ch.isalnum() else '\\%s' % ch
# Not yet.
# N /<CR> (motion) repeat last search, in the forward direction

# 0 (motion) to first character in the line (also: <Home> key)
# ^ (motion) go to first non-blank character in the line
# % (motion) find the next brace, bracket, comment,
#            or "#if"/ "#else"/"#endif" in this line and go to its match
plain_motion_chars = '0^%'
plain_motion =   '|'.join([escape(ch) for ch in plain_motion_chars])
# N + (motion) down N lines, on the first non-blank character (also: CTRL-M and <CR>)
# N _ (motion) down N-1 lines, on the first non-blank character
# N - (motion) up N lines, on the first non-blank character
# N , (motion) repeat the last "f", "F", "t", or "T" N times in opposite direction
# N ; (motion) repeat the last "f", "F", "t", or "T" N times
# N ( (motion) N sentences backward
# N ) (motion) N sentences forward
# N { (motion) N paragraphs backward
# N } (motion) N paragraphs forward
# N | (motion) to column N (default: 1)
# N $ (motion) go to the last character in the line (N-1 lines lower) (also: <End> key)
# N % (motion) goto line N percentage down in the file.  N must be given, otherwise it is the % command.
# N # (motion) search backward for the identifier under the cursor
# N * (motion) search forward for the identifier under the cursor
n_motion_chars = '+_-,;(){}|$%#*'
n_motion_alts = ' | '.join([escape(ch) for ch in n_motion_chars])
# N [#  (motion) N times back to unclosed "#if" or "#else"
# N [(  (motion) N times back to unclosed '('
# N [*  (motion) N times back to start of comment "/*"
# N [[  (motion) N sections backward, at start of section
# N []  (motion) N sections backward, at end of section
# N [{  (motion) N times back to unclosed '{'
open_bracket_chars  = ['[%s' % (ch) for ch in '#(*[]{']
# N ]#  (motion) N times forward to unclosed "#else" or "#endif"
# N ])  (motion) N times forward to unclosed ')'
# N ]*  (motion) N times forward to end of comment "*/"
# N ][  (motion) N sections forward, at end of section
# N ]]  (motion) N sections forward, at start of section
# N ]}  (motion) N times forward to unclosed '}'
close_bracket_chars = [']%s' % (ch) for ch in '#)*[]}']
bracket_chars = open_bracket_chars + close_bracket_chars
bracket_motion = ' | '.join(['%s%s' % (escape(s[0]),escape(s[1])) for s in bracket_chars])
bracket_alts =      r'(?P<bracket_alt>%s)' % (bracket_motion)
# print(bracket_alts)
# gD (motion) goto global declaration of identifier under the cursor
# gd (motion) goto local declaration of identifier under the cursor
g_bare_alts =   r'(?P<bare_g_alts>(gD|gd))'
# N g^      (motion) to first non-blank character in screen line (differs from "^" when lines wrap)
# N g#      (motion) like "#", but also find partial matches
# N g$      (motion) to last character in screen line (differs from "$" when lines wrap)
# N g*      (motion) like "*", but also find partial matches
# N g0      (motion) to first character in screen line (differs from "0" when lines wrap)
# N gE      (motion) backward to the end of the Nth blank-separated WORD
# N ge      (motion) backward to the end of the Nth word
# N gg      (motion) goto line N (default: first line), on the first non-blank character
# N gj      (motion) down N screen lines (differs from "j" when line wraps)
# N gk      (motion) up N screen lines (differs from "k" when line wraps)
g_alt_chars =   ' | '.join([escape(ch) for ch in '^#$*0Eegjk'])
g_alts =        r'(?P<g_n>[0-9]*)(?P<g_alt>g(%s))' % (g_alt_chars)
g_motion = 'g(%s | %s)' % (g_bare_alts,g_alts)
print(g_motion)
.. @+node:ekr.20161117164321.282: *6* @@test vim motion
@language python
# g.cls()

# Unknown:
# N   H  (motion?) go to the Nth line in the window, on the first non-blank
# N   J  (motion?) join N-1 lines (delete newlines)
# VIS J  (motion?) join the highlighted lines
    # M  (motion?) go to the middle line in the window, on the first non-blank
# N   L  (motion?) go to the Nth line from the bottom, on the first non-blank
# o      (motion?) exchange cursor position with start of highlighting

# Not used:
# N %    goto line N percentage down in the file.
#        N must be given, otherwise it is the % command.

#   0    to first character in the line (also: <Home> key)
#   ^    go to first non-blank character in the line
#   %    find the next brace, bracket, comment,
#        or "#if"/ "#else"/"#endif" in this line and go to its match
# N +    down N lines, on the first non-blank character (also: CTRL-M and <CR>)
# N _    down N-1 lines, on the first non-blank character
# N -    up N lines, on the first non-blank character
# N ,    repeat the last "f", "F", "t", or "T" N times in opposite direction
# N ;    repeat the last "f", "F", "t", or "T" N times
# N (    N sentences backward
# N )    N sentences forward
# N {    N paragraphs backward
# N }    N paragraphs forward
# N |    to column N (default: 1)
# N $    go to the last character in the line (N-1 lines lower) (also: <End> key)
# N #    search backward for the identifier under the cursor
# N *    search forward  for the identifier under the cursor
# N B    N blank-separated WORDS backward
# N E    forward to the end of the Nth blank-separated WORD
# N G    goto line N (default: last line), on the first non-blank character
# N N    repeat last search, in opposite direction
# N W    N blank-separated WORDS forward
# N b    N words backward
# N e    forward to the end of the Nth word
# N h    left (also: CTRL-H, <BS>, or <Left> key)
# N j    down N lines (also: CTRL-J, CTRL-N, <NL>, and <Down>)
# N k    up N lines (also: CTRL-P and <Up>)
# N l    right (also: <Space> or <Right> key)
# N n    repeat last search
# N w    N words forward
single_char_motions = [ch for ch in '0^%_+-,;(){}|$#*BEGNWbehjklnw']
# N [#   N times back to unclosed "#if" or "#else"
# N [(   N times back to unclosed '('
# N [*   N times back to start of comment "/*"
# N [[   N sections backward, at start of section
# N []   N sections backward, at end of section
# N [{   N times back to unclosed '{'
m1  = ['['+ ch for ch in '#(*[]{']
# N ]#   N times forward to unclosed "#else" or "#endif"
# N ])   N times forward to unclosed ')'
# N ]*   N times forward to end of comment "*/"
# N ][   N sections forward, at end of section
# N ]]   N sections forward, at start of section
# N ]}   N times forward to unclosed '}'
m2 = [']'+ch for ch in '#)*[]}']
#   gD   goto global declaration of identifier under the cursor
#   gd   goto local declaration of identifier under the cursor
# N g^   to first non-blank character in screen line (differs from "^" when lines wrap)
# N g#   like "#", but also find partial matches
# N g$   to last character in screen line (differs from "$" when lines wrap)
# N g*   like "*", but also find partial matches
# N g0   to first character in screen line (differs from "0" when lines wrap)
# N gE   backward to the end of the Nth blank-separated WORD
# N ge   backward to the end of the Nth word
# N gg   goto line N (default: first line), on the first non-blank character
# N gj   down N screen lines (differs from "j" when line wraps)
# N gk   up N screen lines (differs from "k" when line wraps)
m3 = ['g'+ch for ch in '^#$*0DEdegjk']
# N /<CR>  repeat last search, in the forward direction
m4 = ['/\\n',]
# N F<char>  to the Nth occurrence of <char> to the left
# N T<char>  till before the Nth occurrence of <char> to the left
# N f<char>  to the Nth occurrence of <char> to the right
# N t<char>  till before the Nth occurrence of <char> to the right
char_motions = [ch for ch in 'FTft']
multi_char_leadins = '/g[]'
multi_char_motions = m1+m2+m3+m4
print('\n'.join(single_char_motions))
print('\n'.join(multi_char_motions))
print('\n'.join(['%s<char>' % (ch) for ch in char_motions]))
.. @+node:ekr.20161117164321.283: *5* @test h middle of line
c.testManager.runVimTest(p)

.. @+node:ekr.20161117164321.284: *6* work
first line
.. @+node:ekr.20161117164321.285: *6* before sel=1.5,1.5
first line
.. @+node:ekr.20161117164321.286: *6* after sel=1.4,1.4
first line
.. @+node:ekr.20161117164321.287: *5* @test vr.exec_
import leo.core.leoVim as leoVim
if 0: # When running from leoPy.leo
    import imp
    imp.reload(leoVim)
vc = leoVim.VimCommands(c)
table = (
    # 'gg','gk','#','dd','d3j',
    'h', # works
    # 'l', # works
    # 'j', # Not yet.
    # 'ggg',
)
for s in table:
    status,n1,command,n2,motion = vc.scan(s)
    # print('status',status,'command',command)
    if status == 'done':
        vc.exec_(command,n1,n2,motion)
    else:
        print('status: %s %s' % (status,s))
        vc.command = s
        vc.n1 = n1
        vc.n2 = n2
        vc.motion = motion
        vc.oops()

if g.unitTesting:
    # Unit testing messes up the focus.
    vc.runAtIdle(c.bodyWantsFocusNow)
.. @+node:ekr.20161117164321.288: *5* @test vr.scan
import leo.core.leoVim as leoVim
if 0: # When running from leoPy.leo
    import imp
    imp.reload(leoVim)
import time
trace = False
trace_time = False
<< define test tables >>
vc = leoVim.VimCommands(c)
test_table = (
    ('done',complete_table),
    ('scan',incomplete_table),
    ('oops',error_table),
)
if trace_time: t1 = time.clock()
n = 0
for i in range(1):
    for expected,table in test_table:
        for s in table:
            if table == complete_table:
                command = s
                for expected,command2 in vc.simulate_typing(command):
                    status,n1,command3,n2,motion = vc.scan(command2)
                    n += 1
                    if trace:
                        err = '   ' if status == expected else '***'
                        print('%s%s %s' % (err,status,command2))
                    else:
                        assert status == expected,'expected %s, got %s command: %s' % (
                        expected,status,command2)
            else:
                for prefix in ('','1023456789'):
                    command = prefix + s
                    status,n1,command2,n2,motion = vc.scan(command)
                    n += 1
                    if trace:
                        err = '   ' if status == expected else '***'
                        print('%s%s %s' % (err,status,command))
                    else:
                        assert status == expected,'expected %s, got %s command: %s' % (
                        expected,status,command)
if trace_time:
    delta = time.clock()-t1
    print("%s %6.6f sec." % (n,delta/n))

.. @+node:ekr.20161117164321.289: *6* << define test tables >>
# To do: handle d2d, 2dd, etc.
if 0: # Individual test:
    complete_table = ('ta',)
        # Note: gu is complete, so gu[] is an invalid test.
    incomplete_table = () # 'd3','d4t','dt',
    error_table = ()
else:
    complete_table = (
        '0',
        'N',
        '#',
        'gg','gk','dd',
        'd3j',
        '2dta', # d is not (yet) a motion
        'dFb',
        'gu',
        'g[]',
        'ta',
        't!',
    )
    incomplete_table = (
        'g',
        '[',
        ']',
        '25',
        'd3t',
        'd3',
    )
    error_table = (
        'gX','ZA',
    )
.. @+node:ekr.20161117164321.290: *4* @ignore mini tests
.. @+node:ekr.20161117164321.291: *5* 2to3 script
import os

def run(files):
    args = [r'python c:\python26\Tools\Scripts\2to3.py']
    for z in files:
        args.append(z)
        # args.append('-xprint')
    args.append('>out2')
    args = ','.join(args)
    os.system(args)

tkPass = (
    'EditAttributes','Library',
    'URLloader','UniversalScrolling','UASearch',
    'autotrees','chapter_hoist','cleo','dump_globals',
    'expfolder','geotag','graphed','groupOperations',
    'hoist','import_cisco_config',
    'keybindings','leoupdate',
    'maximizeNewWindows', 'mnplugins','mod_labels',
    'mod_read_dir_outline','mod_tempfname','multifile',
    'newButtons','nodeActions','nodenavigator',
    'open_with','pie_menus','pluginsTest',
    'read_only_nodes','rClick',
    'scheduler','searchbar','searchbox','shortcut_button',
    'script_io_to_body','searchbox',
    'templates','textnode','tkGui','toolbar',
    'xcc_nodes',
)

passList = (
    '__init__','FileActions','UNL',
    'active_path','add_directives','attrib_edit',
    'backlink','base64Packager','baseNativeTree','bibtex','bookmarks',
    'codewisecompleter','colorize_headlines','contextmenu',
    'ctagscompleter','cursesGui','datenodes','debugger_pudb',
    'detect_urls','dtest','empty_leo_file','enable_gc','initinclass',
    'leo_to_html','leo_interface','leo_pdf','leo_to_rtf',
    'leoOPML','leoremote','lineNumbers',
    'macros','mime','mod_autosave','mod_framesize','mod_leo2ascd',
    'mod_scripting','mod_speedups','mod_timestamp',
    'nav_buttons','nav_qt','niceNosent','nodeActions','nodebar',
    'open_shell','outline_export','quit_leo',
    'paste_as_headlines','plugins_menu','pretty_print','projectwizard',
    'qt_main','qt_quicksearch','qtframecommands',
    'quickMove',
        # Warning: changed this line by guessing!
        # func = types.MethodType(func, quickMove)
    'quicksearch','redirect_to_log','rClickBasePluginClasses',
    'run_nodes', # Changed thread.allocate_lock to threading.lock().acquire()
    'rst3',
    'scrolledmessage','setHomeDirectory','slideshow','spydershell','startfile',
    'testRegisterCommand','todo','trace_gc_plugin','trace_keys','trace_tags',
    'vim','xemacs',
)
core_files = (
    'leoApp','leoAtFile','leoCache','leoChapters','leoCommands',
    'leoEditCommands','leoFileCommands','leoFind','leoFrame',
    'leoGlobals','leoGui','leoImport','leoMenu','leoNodes',
    'leoPlugins','leoShadow','leoTangle','leoUndo',
)
external_files = (
    'ipy_leo','lproto',
)
table = (
    ('plugins',passList),
    ('plugins',tkPass),
    ('core',core_files),
    ('external',external_files),
)
files = []
for theDir,aList in table:
    for z in aList:
        if not z.endswith('.py'): z = z + '.py'
        # print(z)
        fn = os.path.abspath(os.path.join('leo',theDir,z))
        if os.path.exists(fn): files.append(fn)
        else: print('*** file not found:',fn)

run(files)
print('done: results are in out2')
.. @+node:ekr.20161117164321.292: *5* Import all plugins script
import glob,os

tkPass = (
    'EditAttributes','Library',
    'URLloader','UniversalScrolling','UASearch',
    'autotrees','chapter_hoist','cleo','dump_globals',
    'expfolder','geotag','graphed','groupOperations',
    'hoist','import_cisco_config',
    'keybindings','leoupdate',
    'maximizeNewWindows', 'mnplugins','mod_labels',
    'mod_read_dir_outline','mod_tempfname','multifile',
    'newButtons','nodeActions','nodenavigator',
    'open_with','pie_menus','pluginsTest',
    'read_only_nodes','rClick',
    'scheduler','searchbar','searchbox','shortcut_button',
    'script_io_to_body','searchbox',
    'templates','textnode','tkGui','toolbar',
    'xcc_nodes',
)
tkPassWithProblems = (
    'at_view', # at_view plugin not loaded: win32Clipboard not present.
    'image', # can not import ImageTk.
    'table', # failed to import 'tktable'
    'xsltWithNodes', # Can not import Ft from plugin leo.plugins.xsltWithNodes.
)
tkFail = (
    'ConceptualSort','at_produce','autocompleter','rowcol',
)
passList = (
    '__init__','FileActions','UNL',
    'active_path','add_directives','attrib_edit',
    'backlink','base64Packager','baseNativeTree','bibtex','bookmarks',
    'codewisecompleter','colorize_headlines','contextmenu',
    'ctagscompleter','cursesGui','datenodes','debugger_pudb',
    'detect_urls','dtest','empty_leo_file','enable_gc','initinclass',
    'leo_to_html','leo_interface','leo_pdf','leo_to_rtf',
    'leoOPML','leoremote','lineNumbers',
    'macros','mime','mod_autosave','mod_framesize','mod_leo2ascd',
    'mod_scripting','mod_speedups','mod_timestamp',
    'nav_buttons','nav_qt','niceNosent','nodeActions','nodebar',
    'open_shell','outline_export','quit_leo',
    'paste_as_headlines','plugins_menu','pretty_print','projectwizard',
    'qt_main','qt_quicksearch','qtframecommands',
    'quickMove',
        # Warning: changed this line by guessing!
        # func = types.MethodType(func, quickMove)
    'quicksearch','redirect_to_log','rClickBasePluginClasses',
    'run_nodes', # Changed thread.allocate_lock to threading.lock().acquire()
    'rst3',
    'scrolledmessage','setHomeDirectory','slideshow','spydershell','startfile',
    'testRegisterCommand','todo','trace_gc_plugin','trace_keys','trace_tags',
    'vim','xemacs',
)
passWithImportProblems = ( # Other than tk input problems.
    'ipython','word_export',
)
dead = (
    'at_folder','exampleTemacsExtension','ironPythonGui','LeoN',
    'rst2','swing_gui','temacs','usetemacs','wxGui',)
error = ( # Real errors with tracebacks.
)
fail = (
    'stickynotes_plus', # requires markdown.
    'zenity_file_dialogs', # requires zenity, and probably ubuntu.
)
noAttribute = (
    # AttributeError: 'module' object has no attribute <module name>
    # This was a sign of a missing init top-level function.
)
changed = (
    'LeoN',
)
plugins = g.os_path_abspath(g.os_path_join(
    g.app.loadDir,'..','plugins','*.py'))
files = glob.glob(plugins)
files.sort()
os.system('cls') # Clear the screen on windows.
for fn in files:
    m = g.shortFileName(fn)[:-3]
    # Change the next line to choose different plugins.
    if m in passList:
        try:
            __import__('leo.plugins.%s' % m)
            if 1: print('pass %s' % m)
        except ImportError:
            if 1: print('FAIL %s' % m)
        except Exception:
            if 1: g.es_exception()
            if 1: print('error %s' % m)
.. @+node:ekr.20161117164321.293: *5* Manual tests...
.. @+node:ekr.20161117164321.294: *6* Other Reformat Paragraph tests
@language plain
@pagewidth 40

A one-line paragraph one two three four five six seven eight nine ten...

An @rawfile tree is a tree whose root headline starts with
@rawfile <filename>. Similarly, an @silentfile tree is a
tree whose root headline starts with an @silentfile
<filename> directive.

    Leo creates derived files from @rawfile and @silentfile trees by writing the body text of all nodes of the tree in outline order.  Leo writes the body text _as is_, without recognizing section definitions, without expanding section references, and without treating directives specially in any way.  In particular, Leo copies all directives, including @space or @c directives, to the derived file as text. Exception: Leo recognizes the @ignore directive in @rawfile or @silentfile nodes, so you may use the @ignore directive as usual to prevent Leo from writing @rawfile and @silentfile trees.

There are several difference between @rawfile and @silentfile trees:

  This
  is
  a
  test.

  1. This is the first line and it is really really really long. And it has
     a hanging indentation.
     and another line.

  2. This is a lllllllllllllllllllllllllllllllllllloooooooooooooooooong
     next item.
     And it too has a hanging indentation.

  3. This is an exxxxxxxxxxxxxxxxxxxxxxxxtrrrrrrrrrrrrrrreeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeemlylong
     word.
And it too has a hanging indentation.
.. @+node:ekr.20161117164321.295: *6* Test of @tabwidth
@tabwidth -4
@language plain


    a   b   c
a   b   c   d
aa  b   c   d
aaa b   c   d
end
.. @+node:ekr.20161117164321.296: *6* Test of pasting into big node
@killcolor
@language plain

Note: Previously, one could crash Leo by pasting a large text into a headline.  Leo now truncates that text, and furthermore Leo no longer makes all headline text into one gigantic line.  Therefore, we don't have to test Tk's ability to handle super-long lines.

The test:  Copy the following and paste it into a headline.  Leo should give 2 truncation messages:

- Truncating headline to one line.
- Truncating headline to 250 characters.

About a year ago I found the website at www.literateprogamming.com and was immediately convinced that the basic idea of Literate Programming is an important breakthrough. At the time I was working on a contract trying to decipher a true masterpiece of over-engineering, and if at any time during the construction of this masterpiece the perpetrators had been required to explain themselves in English, my client would have saved millions of dollars.

I never did try CWEB or NOWEB though, because on the literate programming site I read about a tool named “Leo” that combined outlines with Literate Programming techniques. Since I’ve always found outlining tools very useful I downloaded and tried this. I found that using this tool completely changed my programming practice and brought out all of the power inherent in the original Literate Programming idea as I understood it.
.. @+node:ekr.20161117164321.297: *6* Test of Remove sentinels
import os

g.pr(os.getcwd())
name = g.os_path_join("test","removeSentTest.txt")
c.importCommands.removeSentinelsCommand(name)
.. @+node:ekr.20161117164321.298: *6* Test new docutils stull
import glob

g.pr('-' * 40)

tm = c.testManager

if 0:
    g.pr("modules in test.leo...")
    paths = tm.findAllAtFileNodes(c)
    modules = tm.importAllModulesInPathList(paths)
    for module in modules:
        g.pr(module)

if 1:
    g.pr("modules in leo/src...")
    path = g.os_path_join(g.app.loadDir,"..","src")
    modules = tm.importAllModulesInPath(path)
    for module in modules:
        g.pr(module)

if 0:
    directory = g.os_path_join(g.app.loadDir,"..","src")
    glob_path = g.os_path_join(directory,"leo*.py")
    files = glob.glob(glob_path)
    modules = tm.importAllModulesInPathList(files)
    for module in modules:
        g.pr(module)
.. @+node:ekr.20161117164321.299: *6* Manual test of TM.replaceOutline
outline1 = p.firstChild()
outline2 = outline1.next()
assert(outline1.h=="outline1")
assert(outline2.h=="outline2")

c.testManager.replaceOutline(outline1,outline2)
c.redraw()
c.checkOutline()
.. @+node:ekr.20161117164321.300: *7* outline1
.. @+node:ekr.20161117164321.301: *8* a
.. @+node:ekr.20161117164321.302: *7* outline2
.. @+node:ekr.20161117164321.303: *8* b
.. @+node:ekr.20161117164321.304: *5* Mini test arguments to hooks
"""Mini test that documentation of hooks in leoDocs.leo is correct.

hookData should match that documentation for this test to be effective.

This is not a complete unit test:  it does not force executions of all hooks.
"""

<< imports >>
<< define hookData >>
<< define typeData >>
checked = [] # List of all hooks that have been checked.

@others

tags = [] 
for name,args in hookData:
    tags.append(name)
    << define checkHook >>
    leoPlugins.registerHandler(name,checkHook)

if 0: # print all hooks.
    handlers = leoPlugins.getHandlersForTag(tags)
    if handlers:
        g.pr("-" * 20)
        for h in handlers:
            g.pr(h)
.. @+node:ekr.20161117164321.305: *6* << imports >>
import leoColor
import leoCommands
import leoNodes
import leoPlugins
import leoTkinterTree

import types
import Tkinter as Tk
.. @+node:ekr.20161117164321.306: *6* << define hookData >>
hookData = (
    ("bodyclick1",   ("c","p","v","event")),
    ("bodyclick2",   ("c","p","v","event")),
    ("bodydclick1",  ("c","p","v","event")),
    ("bodydclick2",  ("c","p","v","event")),
    ("bodykey1",     ("c","p","v","ch","oldSel","undoType")),
    ("bodykey2",     ("c","p","v","ch","oldSel","undoType")),
    ("bodyrclick1",  ("c","p","v","event")),
    ("bodyrclick2",  ("c","p","v","event")),
    ("boxclick1",    ("c","p","v","event")),
    ("boxclick2",    ("c","p","v","event")),
    ("command1",     ("c","p","v","label")),
    ("command2",     ("c","p","v","label")),
    ("drag1",        ("c","p","v","event")),
    ("drag2",        ("c","p","v","event")),
    ("dragging1",    ("c","p","v","event")),
    ("dragging2",    ("c","p","v","event")),
    ("end1",         None),
    ("enddrag1",     ("c","p","v","event")),
    ("enddrag2",     ("c","p","v","event")),
    ("headclick1",   ("c","p","v","event")),
    ("headclick2",   ("c","p","v","event")),
    ("headrclick1",  ("c","p","v","event")),
    ("headrclick2",  ("c","p","v","event")),
    ("headkey1",     ("c","p","v","ch")),
    ("headkey2",     ("c","p","v","ch")),
    ("hypercclick1", ("c","p","v","event")),
    ("hypercclick2", ("c","p","v","event")),
    ("hyperenter1",  ("c","p","v","event")),
    ("hyperenter2",  ("c","p","v","event")),
    ("hyperleave1",  ("c","p","v","event")),
    ("hyperleave2",  ("c","p","v","event")),
    ("iconclick1",   ("c","p","v","event")),
    ("iconclick2",   ("c","p","v","event")),
    ("iconrclick1",  ("c","p","v","event")),
    ("iconrclick2",  ("c","p","v","event")),
    ("icondclick1",  ("c","p","v","event")),
    ("icondclick2",  ("c","p","v","event")),
    ("idle",         ("c",)),
    ("menu1",        ("c","p","v")),
    ("menu2",        ("c","p","v")),
    ("open1",        ("old_c","new_c","fileName")),
    ("open2",        ("old_c","new_c","fileName")),
    ("openwith1",    ("c","p","v","openType","arg","ext")),
    ("openwith2",    ("c","p","v","openType","arg,ext" )),
    ("recentfiles1", ("c","p","v","fileName","closeFlag")),
    ("recentfiles2", ("c","p","v","fileName","closeFlag")),
    ("save1",        ("c","p","v","fileName" )),
    ("save2",        ("c","p","v","fileName" )),
    ("select1",      ("c","new_p","old_p","new_v","old_v")),
    ("select2",      ("c","new_p","old_p","new_v","old_v")),
    ("select3",      ("c","new_p","old_p","new_v","old_v")),
    ("set-mark",     ("c","p","v")),
    ("start1",       None),
    ("start2",       ("c","p","v","fileName" )),
    ("unselect1",    ("c","new_p","old_p","new_v","old_v")),
    ("unselect2",    ("c","new_p","old_p","new_v","old_v")),
    ("@url1",        ("c","p","v")),
    ("@url2",        ("c","p","v")),
    # Stub hooks.
    ("after-redraw-outline",         ("c",)),
    ("clear-mark",                   ("c","p","v")),
    ("close-frame",                  ("c",)),
    ("color-optional-markup",        ("colorer","p","v","s","i","j","colortag")),
    ("create-optional-menus",        ("c",)),
    ("destroy-all-global-windows",   None),
    ("draw-outline-box",             ("tree","p","v","x","y")), #
    ("draw-outline-icon",            ("tree","p","v","x","y")), #
    ("draw-outline-node",            ("tree","p","v","x","y")), #
    ("draw-outline-text-box",        ("tree","p","v","x","y")), #
    ("create-popup-menu-items",      ("c","p","v","event")),
    ("enable-popup-menu-items",      ("c","p","v","event")),
    ("init-color-markup",            ("colorer","p","v")),
    ("new",                          ("old_c","new_c")),
    ("redraw-entire-outline",        ("c",)),
    ("scan-directives",              ("c","p","v","s","old_dict","dict","pluginsList")),
    ("set-mark",                     ("c","p","v" )),
    ("show-popup-menu",              ("c","p","v","event")),
)
.. @+node:ekr.20161117164321.307: *6* << define typeData >>
typeData = {
    "arg":      types.StringType,
    "c":        leoCommands.Commands,
    "ch":       types.StringType,
    "closeFlag":types.StringType,
    "colorer":  leoColor.colorizer,
    "colortag": types.StringType,
    "dict":     types.DictType,
    "event":    Tk.Event,
    "ext":      types.StringType,
    "fileName": types.StringType,
    "i":        types.IntType,
    "j":        types.IntType,
    "label":    types.StringType,
    "new_c":    leoCommands.Commands,
    "new_p":    leoNodes.position,
    "newSel":   types.TupleType,
    "new_v":    leoNodes.position,
    "old_c":    leoCommands.Commands,
    "old_dict": types.DictType,
    "old_p":    leoNodes.position,
    "oldSel":   types.TupleType,
    "old_v":    leoNodes.position,
    "openType": types.StringType,
    "p":        leoNodes.position,
    "pluginsList": types.ListType,
    "s":        types.UnicodeType,
    "tree":     leoTkinterTree.leoTkinterTree,
    "v":        leoNodes.position,
    "undoType": types.StringType,
    "x":        types.IntType,
    "y":        types.IntType,
}
.. @+node:ekr.20161117164321.308: *6* << defineCheckHook >>
def checkHook (tag,keywords,args=args):

    """Check to see that the keywords passed to the hook are as described in args.
    Each arg is a list of strings whose type is defined in typeData."""

    global checked, verbose
    if tag in checked: return
    ok = True
    checked.append(tag)
    if args is None: args = []
    args = list(args)
    args.sort()
    keys = list(keywords.keys())
    keys.sort()

    if len(args) != len(keys):
        g.pr("%25s expected:" % (tag),args)
        g.pr("%25s      got:" % (tag),keys)
        ok = False
    else:
        for arg,key in zip(args,keys):
            arg_type = typeData.get(arg)
            val = keywords.get(key)
            if not checkOneHook(arg_type,val):
                g.pr("%25s      arg:" % (tag), arg)
                g.pr("%25s expected:" % (tag), arg_type)
                g.pr("%25s      got:" % (tag), type(val))
                ok = False
    if ok:
        g.pr(tag)
.. @+node:ekr.20161117164321.309: *6* checkOneHook
def checkOneHook (arg_type, val):

    if 0:
        if arg_type != type(val):
            g.trace(arg,key,arg_type,type(val))

    return (
        (arg_type is type(val)) or
        (arg_type == types.StringType and type(val) is types.UnicodeType) or
        (type(arg_type) == types.ClassType and isinstance(val,arg_type)))
.. @+node:ekr.20161117164321.310: *5* Mini test of add-editor
c.k.simulateCommand('add-editor')
c.k.simulateCommand('delete-editor')
.. @+node:ekr.20161117164321.311: *5* Mini test of g.es_exception
try:
    assert False, 'Assert False'
except AssertionError:
    g.es_exception()
.. @+node:ekr.20161117164321.312: *5* Mini test of g.pdb
# Running this as a unit test would hang the unit tests!
g.pdb()
.. @+node:ekr.20161117164321.313: *5* Mini test of unicode stuff
@first # -*- coding: utf-8 -*-

table = (
    'test',
    'Ä 궯 奠',
    'Ä 궯 奠 after', # fails with cp6501: after is duplicated.
)

print('*'*20)
print('isPython3: %s' % g.isPython3)

for s in table:
    if g.isPython3:
        s = s.encode('ascii','replace') # create bytes.
    g.es(repr(s))
    g.es(s)
    g.pr ('g.pr(s)       : %s' % s)
    g.pr ('g.pr(repr(s)) : %s' % repr(s))
    print('print(s)      : %s' % s)
    print('print(repr(s)): %s' % s)
.. @+node:ekr.20161117164321.314: *5* Mini test that g.es leaves focus unchanged
if g.app.isExternalUnitTest:
    pass # Prints to console, which is annoying.
else:
    old_focus = c.get_focus()
    
    for flag in (False,True):
        g.es('Hi')
        
    if flag:
        c.outerUpdate() # Restores focus, especially when run from a script.
    new_focus = c.get_focus()
    
    try:
        assert old_focus == new_focus,'old focus: %s new focus: %s' % (
            old_focus,new_focus)
    except AssertionError:
        c.bodyWantsFocusNow()
.. @+node:ekr.20161117164321.315: *5* mini test that print-bindings puts results in Bindings tab
if g.app.isExternalUnitTest:
    pass
else:
    c.k.simulateCommand('print-bindings')
    log = c.frame.log
    
    # This works when run via execute-script.
    assert log.tabName == 'Bindings',log.tabName
    
    # w = c.frame.log.contentsDict.get('Bindings')
    # assert w,'no Bindings widget'
    # wrapper = w.leo_log_wrapper
    # s = wrapper.getAllText()
    # assert s,wrapper
.. @+node:ekr.20161117164321.316: *5* Mini tests of script buttons
.. @+node:ekr.20161117164321.317: *6* Redundant: @suite run all doctests in @file nodes
import doctest
import unittest

tm = c.testManager

createUnitTest = True

if createUnitTest:
    suite = unittest.makeSuite(unittest.TestCase)
else:
    suite = None

paths   = tm.findAllAtFileNodes(c)
modules = tm.importAllModulesInPathList(paths)

if createUnitTest:
    suite = tm.createUnitTestsFromDoctests(modules)
else:
    for module in modules:
        doctest.testmod(module,verbose=True,report=False)

if suite:
    g.app.scriptDict['suite'] = suite
.. @+node:ekr.20161117164321.619: *6* runProfile button mini-test
for i in range(10000):
    if i and (i % 1000) == 0:
        g.pr(i)
.. @+node:ekr.20161117164321.620: *6* runTimeit mini-test
i = 0
for i in range(100000):
    i += 1
    i -= 1
.. @+node:ekr.20161117164321.320: *6* profile redraws
# c.redraw just schedules the actual drawing.
# We want to profile the actual idle-time drawing.

c.frame.tree.idle_redraw()
.. @+node:ekr.20161117164321.321: *5* Other tests
@language python
@tabwidth -4
.. @+node:ekr.20161117164321.322: *6* @@nosent test-niceNosent
.. @+node:ekr.20161117164321.323: *7* part 1
part 1, line 1
part 2, line 2, no newline
.. @+node:ekr.20161117164321.324: *7* part 2
part 2, line 1, no newline
.. @+node:ekr.20161117164321.325: *7* part 3
part 3, line 1
part 3, line 2, newline
.. @+node:ekr.20161117164321.326: *6* @nowrap tests
@nowrap
aaaaaaaaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbb cccccccccccccccccccc ddddddddddddddd eeeeeeeeeeeeeee ffffffffffffffff 
.. @+node:ekr.20161117164321.327: *6* Experiments
.. @+node:ekr.20161117164321.328: *7* Test of moving positions
class position:
    def __init__(self):
        self.v = "a"
    def move(self):
        self.v = "b"

p = position()
v = p.v
g.pr("before", v, p.v, v is p.v)
p.move()
g.pr("after ", v, p.v, v is p.v)
.. @+node:ekr.20161117164321.329: *7* Test of using an iterator inside a list comprehension
class test_iter_class:
    def __init__ (self):
        self.vals = ("a","b","c")
        self.n = 0
    def __iter__(self):
        return self
    def next(self):
        if self.n < len(self.vals):
            val = self.vals[self.n]
            self.n += 1
            return val
        else:
            raise StopIteration

def test_iter(): return test_iter_class()

vals = [val for val in test_iter()]

g.pr(vals)
.. @+node:ekr.20161117164321.330: *7* Test of using c.allNodes_iter to create a list of all positions
g.pr('-'*20)

# These are equivalent.
positions1 = [p for p in c.allNodes_iter(copy=True)]
positions2 = [p.copy() for p in c.allNodes_iter()]

assert(len(positions1) == len(positions2))
for i in range(len(positions1)):
    assert(positions1[i] == positions2[i])

if 0:
    for p in positions1:
        g.pr(p)
g.pr("done")
.. @+node:ekr.20161117164321.331: *7* Creating a list of distinct vnodes
g.pr('-'*20)

positions = [p.copy() for p in c.allNodes_iter()]

tnodes = {} ; vnodes = []
for p in c.allNodes_iter():
    t = p.v.t
    if tnodes.get(t) is None:
        tnodes[t]=t
        vnodes.append(p.v)

g.pr(len(positions),len(vnodes))

for v in vnodes:
    g.pr(v)
.. @+node:ekr.20161117164321.332: *7* test of list comparisons
stack1 = ["a","b","c"]
stack2 = ["a","b","c"]
stack3 = ["a","b","d"]
stack4 = ["a","b"]
g.pr(stack1 == stack2)
g.pr(stack1 == stack3)
g.pr(stack1 == stack4)
.. @+node:ekr.20161117164321.333: *7* test that childIndex doesn't mess with p
g.pr(p.h)
g.pr(p.childIndex())
g.pr(p.h)
.. @+node:ekr.20161117164321.334: *7* Test of __cmp__ vrs equal
import timeit

s1 = '''\
class test(object):
    def __cmp__(self,p2):   return 0
    def equal(self,p2):     return 0
p1 = test() ; p2 = test()'''

s2 = '''\
class test:
    def __cmp__(self,p2):   return 0
    def equal(self,p2):     return 0
p1 = test() ; p2 = test()'''

s3 = '''\
import leoNodes
p1 = leoNodes.position(None,[])
p2 = leoNodes.position(None,[])'''

for s in (s1,s2,s3):
    t1 = timeit.Timer(stmt='p1==p2',setup=s).timeit()
    t2 = timeit.Timer(stmt='p1.equal(p2)',setup=s).timeit()
    g.pr("%2.2f,%2.2f,%0.2f" % (t1,t2,t1/t2))
.. @+node:ekr.20161117164321.335: *7* Test print
# "LPT1:", "PRN:" and "PRN" all freeze

s = 'stuff\n'
port = 'USB002'

try:
    f = file(port,'w')
    f.write(s)
    f.flush()
    f.close()
    g.pr("done")
except IOError:
    g.pr("Can not open",port)
.. @+node:ekr.20161117164321.336: *7* String-based imports...
@ By far the simplest way is just to write the string to a temp file, then import the temp files.

All other approaches quickly get deeply involved with Leo's internals...
.. @+node:ekr.20161117164321.337: *8* import from string
@language plain

The first idea was to use Python's imp module to simulate an import from a file.  This does not work well because imp expects a file, not a StringIO object.

The second idea was to use Python's parser module.  But this returns an instance type, not a module.

A third idea would be to subclass the file type to fool the imp module.

A fourth idea would be to use the ihooks module.  Apparently this module was designed to do something like what I am trying to do!  However, there doesn't seem to be docs for it, so I have imported the code...

@color
.. @+node:ekr.20161117164321.338: *9* @@test import from string
import imp
import StringIO

@
load_module( name, file, filename, description) 

Load a module that was previously found by find_module() (or by an otherwise conducted search yielding compatible results). This function does more than importing the module: if the module was already imported, it is equivalent to a reload()! The name argument indicates the full module name (including the package name, if this is a submodule of a package). The file argument is an open file, and filename is the corresponding file name; these can be None and '', respectively, when the module is not being loaded from a file. The description argument is a tuple, as would be returned by get_suffixes(), describing what kind of module must be loaded. 
If the load is successful, the return value is the module object; otherwise, an exception (usually ImportError) is raised. 

Important: the caller is responsible for closing the file argument, if it was not None, even when an exception is raised. This is best done using a try ... finally statement.
@c

s = """

def foobar(): pass

"""

@ get_suffixes( ) 

Return a list of triples, each describing a particular type of module. Each triple has the form (suffix, mode, type), where suffix is a string to be appended to the module name to form the filename to search for, mode is the mode string to pass to the built-in open() function to open the file (this can be 'r' for text files or 'rb' for binary files), and type is the file type, which has one of the values PY_SOURCE, PY_COMPILED, or C_EXTENSION, described below.
@c

g.pr('-' * 20)
description = (".py","r",imp.PY_SOURCE)
theFile = StringIO.StringIO(s) # Create a file-like object
g.pr(repr(theFile))
try:
    imp.load_module("myModule",theFile,"myFileName",description)
except:
    g.es_exception()



.. @+node:ekr.20161117164321.339: *8* Subclass the file type for use with imp module
if 0:
    class myFile(file):
        pass

    g.pr(myFile)
    g.pr(issubclass(myFile,file))
    g.pr(isinstance(myFile,file))
    g.pr(super(myFile))
    g.pr(__import__)

if 0:
    old_import = __import__

    def myImport(*args,**keys):
        g.pr("myImport")
        global old_import
        old_import(*args,**keys)

    __import__ = myImport

mod = __import__("leoApp")
g.pr(mod)
.. @+node:ekr.20161117164321.340: *8* Use parser module to simulate import from string
import compiler

for child in p.children_iter():
    h = child.h
    body = child.b

    try:
        val = compiler.parse(body)
        g.pr(type(val))
        g.pr(val)
    except SyntaxError:
        g.es("Syntax error: %s" % h,color="blue")
.. @+node:ekr.20161117164321.341: *9* test1
import doctest
g.pr(doctest)
.. @+node:ekr.20161117164321.342: *6* Make sure openWith changes are benign
arg = "arg" ; filename = "fileName"
path = "path" ; shortPath = "shortPath"
vtuple = "vtuple"

def test(a,b):
    assert(a==b)

test(
    "os.system("+arg+shortPath+")",
    "os.system(%s)" % (arg+shortPath))
test(
    "os.startfile("+arg+shortPath+")",
    "os.startfile(%s)" % (arg+shortPath))
# test(
    # "exec("+arg+shortPath+")",
    # "exec(%s)" % (arg+shortPath))
test(
    "os.spawnl("+arg+","+filename+','+ shortPath+")",
    "os.spawnl(%s,%s,%s)" % (arg,filename,shortPath))
test(
    "os.spawnv("+arg[0]+","+repr(vtuple)+")",
    "os.spawnv(%s,%s)" % (arg[0],repr(vtuple)))
.. @+node:ekr.20161117164321.343: *6* Perfect import stuff...
@language python
@tabwidth -4
.. @+node:ekr.20161117164321.344: *7* Mulder Update script
# EKR: I don't remember the status of this.

@language python

import shutil

testing = True
sourcedir=r"c:/prog/test/perfectImport"
targetdir=r"c:/prog/test/perfectImport/leo"
s1 = g.os_path_join(sourcedir,"leoAtFile.py")
t1 = g.os_path_join(targetdir,"leoAtFile.py")
files = [(s1,t1)]

@others

g.pr('\n' + '-' * 20)
sync(files) # push or pull, depending on date.
.. @+node:ekr.20161117164321.345: *8* sync
def sync(files):

    """Do a pull or a push, depending on the date of the files."""

    none, push, pull = 'None', 'push', 'pull'
    mu = g.mulderUpdateAlgorithm()

    for sourcefilename, targetfilename in files:
        << compute sourcetime and targettime >>
        << compute operation >>
        if operation == push:
            if testing: g.pr(push, sourcefilename, targetfilename)
            strippedLines = mu.removeSentinelsFromFile(sourcefilename)
            mu.write_if_changed(strippedLines,sourcefilename,targetfilename)
            mu.copy_time(sourcefilename,targetfilename)
        elif operation == pull:
            if testing: g.pr(pull, sourcefilename, targetfilename)
            if sourcetime:
                mu.propagateDiffsToSentinelsFile(sourcefilename,targetfilename)
                mu.copy_time(targetfilename,sourcefilename)
            else:
                shutil.copy2(targetfilename,sourcefilename)
.. @+node:ekr.20161117164321.346: *9* << compute sourcetime and targettime >>
sourcetime = targettime = None

if g.os_path_exists(sourcefilename):
    sourcetime = g.os_path_getmtime(sourcefilename)

if g.os_path_exists(targetfilename):
    targettime = g.os_path_getmtime(targetfilename)
.. @+node:ekr.20161117164321.347: *9* << compute operation >>
operation = None
if sourcetime:
    if targettime:
        if sourcetime > targettime:
            operation = push
        elif sourcetime < targettime:
            operation = pull
    else:
        operation = push
elif targettime:
    operation = pull
.. @+node:ekr.20161117164321.348: *7* Perfect Import Script
# Run this script to import a file.
# This is undoable because the Import @file command is undoable.

path = r"c:\prog\test\perfectImport"

# Two files from Python23/Lib
name1 = g.os_path_join(path,"formatter.py")
name2 = g.os_path_join(path,"SimpleHTTPServer.py")
names = [name1]

c.importCommands.importFilesCommand (names,"@file",
    perfectImport=True,testing=False,verbose=True)
.. @+node:ekr.20161117164321.349: *6* Printing tests...
.. @+node:ekr.20161117164321.350: *7* Print findAllPotentiallyDirtyNodes
g.pr('-'*20)

for p in c.allNodes_iter():
    if p.isDirty():
        vnodes = p.findAllPotentiallyDirtyNodes()
        g.pr('-'*5, p)
        for v in vnodes:
            g.pr(v)

g.pr("done")
.. @+node:ekr.20161117164321.351: *7* Print iterations: do not delete
import leoNodes

position = leoNodes.position

@others

current = pos = c.p
child1 = current.firstChild()
child2 = child1.firstChild()

if 0:
    g.pr('-'*10, "parents")
    for p in child2.parents_iter(): g.pr(p)
if 0:
    g.pr('-'*10, "subtree")
    for p in pos.subtree_iter(): g.pr(p)
if 0:
    g.pr('-'*10, "children")
    for p in child1.children_iter(): g.pr(p)
if 0:
    g.pr('-'*10, "siblings")
    for p in pos.siblings_iter(): g.pr(p)
if 1:
    g.pr('-'*10, "all nodes")
    for p in c.allNodes_iter():
        g.pr(p.isCloned(),p)
.. @+node:ekr.20161117164321.367: *8* b
.. @+node:ekr.20161117164321.368: *9* c
.. @+node:ekr.20161117164321.369: *10* c2
.. @+node:ekr.20161117164321.370: *11* c3
.. @+node:ekr.20161117164321.371: *11* c4
.. @+node:ekr.20161117164321.357: *8* Clone test data
.. @+node:ekr.20161117164321.358: *9* aa
.. @+node:ekr.20161117164321.366: *9* a
.. @+node:ekr.20161117164321.367: *10* b
.. @+node:ekr.20161117164321.368: *11* c
.. @+node:ekr.20161117164321.369: *12* c2
.. @+node:ekr.20161117164321.370: *13* c3
.. @+node:ekr.20161117164321.371: *13* c4
.. @+node:ekr.20161117164321.365: *9* d
.. @+node:ekr.20161117164321.366: *10* a
.. @+node:ekr.20161117164321.367: *11* b
.. @+node:ekr.20161117164321.368: *12* c
.. @+node:ekr.20161117164321.369: *13* c2
.. @+node:ekr.20161117164321.370: *14* c3
.. @+node:ekr.20161117164321.371: *14* c4
.. @+node:ekr.20161117164321.372: *9* e
.. @+node:ekr.20161117164321.373: *9* z
.. @+node:ekr.20161117164321.374: *8* last node
.. @+node:ekr.20161117164321.375: *7* Print isAnyAtFileNode
g.pr('-'*20)

for p in c.allNodes_iter():
    if p.isAnyAtFileNode():
        g.pr(p)

g.pr("done")
.. @+node:ekr.20161117164321.376: *7* Print fundChildrenOf and
tm = c.testManager

g.pr("children", '-' * 20)
children = tm.findChildrenOf(p)
for child in children: g.pr(child.h)

g.pr("subtree", '-' * 20)
descendants = tm.findSubnodesOf(p)
for descendant in descendants: g.pr(descendant.h)
.. @+node:ekr.20161117164321.377: *7* Tests of pickle & hexlify
import binascii
import pickle

d = { "a":True }

g.pr('-' * 40)

s = pickle.dumps(d,bin=True)
s2 = binascii.hexlify(s)
g.pr(`s`,s2)

s3 = binascii.unhexlify(s2)
d2 = cPickle.loads(s3)

g.pr(`d2`)
g.pr(d == d2, d is d2)
.. @+node:ekr.20161117164321.378: *7* Test of undo registration
def redoBletch(self):
    g.trace()

def undoBletch(self):
    g.trace()

u = c.undoer

if 0:
    # bad functions
    u.registerUndoHandlers("Bletch","abc","xyz")
else:
    u.registerUndoHandlers("Bletch",undoBletch,redoBletch)

# "Execute" the Bletch command :-)  The Edit command should contain "Undo Bletch"
u.setUndoParams("Bletch",p)

# Selecting "Undo Bletch" will enable "Redo Bletch", etc.
.. @+node:ekr.20161117164321.379: *7* Test of unknownAttributes
d = { "a":True }

if 1:
    # Warning: executing this in the a2 code base will cause any save operation to fail.
    p.v.unknownAttributes = { "myPlugin" : d }

g.pr(repr(p.v.unknownAttributes))
.. @+node:ekr.20161117164321.380: *7* Test of "end1" hook
import leoPlugins

def onEnd (tag,keys):
    g.pr("onEnd",tag,keys)

count = 0

def onIdle (tag,keys):
    global count ; count += 1
    if count % 10 == 0:
        g.pr("onIdle",count,keys.get("c"))

leoPlugins.registerHandler("end1", onEnd)
g.pr("onEnd registered as end1 hook")

leoPlugins.registerHandler("idle", onIdle)
g.pr("onIdle registered as idle hook")
.. @+node:ekr.20161117164321.381: *7* Print timestamps of all nodes
for p in c.all_positions_iter():
    g.pr(p.v.t.fileIndex)
.. @+node:ekr.20161117164321.382: *7* test of focus
g.pr(c.frame.bodyCtrl.focus())
.. @+node:ekr.20161117164321.383: *7* Using a generator instead of readLinesClass
# This kind of code is used in the prototypes of new commands.

from __future__ import generators

@others

lines = "a\nb\nc\nd"

if 1: # Both work
    readline = g.readLinesGenerator(lines).next
else:
    readline = g.readLinesClass(lines).next

g.pr('-' * 20)

if 1: # Both work
    for s in g.readLinesGenerator(lines):
        g.pr(s,)
else:
    while 1:
        s = readline()
        if s: g.pr(s,)
        else: break

g.pr('\n' + '-' * 20)
.. @+node:ekr.20161117164321.384: *7* Test of better error messages in Execute Script command
# Test
@others
# Last
.. @+node:ekr.20161117164321.385: *8* Contains error
a = 1
g.pr("hello")
c = b
.. @+node:ekr.20161117164321.386: *7* print all docstrings from a module
import leoTest
import types

specialDictNames = ('__builtins__','__doc__','__name__','__file__','__module__')

def printDoc(x,s):
    if hasattr(x,"__doc__") and x.__doc__:
        g.pr("%4d %s" % (len(x.__doc__),s))
    else:
        g.pr("%4s %s" % (' ',s))

g.pr('-' * 60)
g.pr("%4d %s" % (len(leoTest.__doc__),"leoTest"))

if 1:
    for s in leoTest.__dict__:
        if s not in specialDictNames:
            x = getattr(leoTest,s)
            if type(x) != types.ModuleType:
                printDoc(x,s)
                if type(x) == types.ClassType:
                    for s2 in x.__dict__:
                        x2 = getattr(x,s2)
                        if s2 not in specialDictNames:
                            g.pr(' '*4,)
                            printDoc(x2,s2)
else:
    << print names sorted by type >>
.. @+node:ekr.20161117164321.387: *8* << print names sorted by type >>
for theType,typeName in (
    (types.ModuleType,"modules"),
    (types.ClassType,"classes"),
    (types.FunctionType,"functions"),
):

    g.pr("\n%s..." % typeName)
    for s in leoTest.__dict__:

        if s not in specialDictNames:
            x = getattr(leoTest,s)
            if type(x) == theType:
                printDoc(x,s)
                if theType == types.ClassType:
                    g.pr("\tmethods...")
                    for s2 in x.__dict__:
                        x2 = getattr(x,s2)
                        if s2 not in specialDictNames:
                            g.pr("\t",newline=False)
                            printDoc(x2,s2)
.. @+node:ekr.20161117164321.388: *6* Registering & unregistering the "new" drawing hooks
.. @+node:ekr.20161117164321.389: *7* Register all new hooks
import leoPlugins as plugins

def traceHook(tag,event):
    g.trace(tag)

tags = (
    "boxclick1","boxclick2",
    "drag1","drag2",
    "dragging1","dragging2",
    "enddrag1","enddrag2",
    "iconclick1","iconclick2"  , 
    "iconrclick1","iconrclick2",
    "icondclick1","icondclick2",
)

plugins.registerHandler(tags,traceHook)

handlers = plugins.getHandlersForTag(tags)
if handlers:
    g.pr("-" * 20)
    for h in handlers:
        g.pr(h)
.. @+node:ekr.20161117164321.390: *7* Unregister all new hooks
import leoPlugins as plugins

tags = (
    "boxclick1","boxclick2",
    "drag1","drag2",
    "dragging1","dragging2",
    "enddrag1","enddrag2",
    "iconclick1","iconclick2"  , 
    "iconrclick1","iconrclick2",
    "icondclick1","icondclick2",
)

for tag in tags:
    handlers = plugins.getHandlersForTag(tag)
    if handlers:
        g.pr(handlers)
        for f in handlers:
            plugins.unregisterHandler(tag,f)

handlers = plugins.getHandlersForTag(tags)
if handlers:
    g.pr("-" * 20)
    for h in handlers:
        g.pr(h)
.. @+node:ekr.20161117164321.391: *7* Print all new hooks
import leoPlugins as plugins

tags = (
    "boxclick1","boxclick2",
    "drag1","drag2",
    "dragging1","dragging2",
    "enddrag1","enddrag2",
    "iconclick1","iconclick2"  , 
    "iconrclick1","iconrclick2",
    "icondclick1","icondclick2",
)

handlers = plugins.getHandlersForTag(tags)
if handlers:
    g.pr("-" * 20)
    for h in handlers:
        g.pr(h)
.. @+node:ekr.20161117164321.392: *6* ReportLab sample scripts
import sys
sys.path.append(r'c:\reportlab_1_20')

debug = True

@others

from reportlab.pdfgen import canvas
c = canvas.Canvas('hello.pdf')
for i in (10,50):
    text(c,'x'*10,i,i)
# pencil(c,text='Note')

key = 'key1'

c.bookmarkPage(key)
c.addOutlineEntry('OutlineEntry',key)


c.showPage()
c.save()
.. @+node:ekr.20161117164321.393: *7* text
def text(c,text,i=100,j=100):
    c.drawString(i,j,text)
.. @+node:ekr.20161117164321.394: *7* pencil
def pencil(canvas, text="No.2"):
    from reportlab.lib.colors import yellow, red, black,white
    from reportlab.lib.units import inch
    u = inch/10.0
    canvas.setStrokeColor(black)
    canvas.setLineWidth(4)
    # draw erasor
    canvas.setFillColor(red)
    canvas.circle(30*u, 5*u, 5*u, stroke=1, fill=1)
    # draw all else but the tip (mainly rectangles with different fills)
    canvas.setFillColor(yellow)
    canvas.rect(10*u,0,20*u,10*u, stroke=1, fill=1)
    canvas.setFillColor(black)
    canvas.rect(23*u,0,8*u,10*u,fill=1)
    canvas.roundRect(14*u, 3.5*u, 8*u, 3*u, 1.5*u, stroke=1, fill=1)
    canvas.setFillColor(white)
    canvas.rect(25*u,u,1.2*u,8*u, fill=1,stroke=0)
    canvas.rect(27.5*u,u,1.2*u,8*u, fill=1, stroke=0)
    canvas.setFont("Times-Roman", 3*u)
    canvas.drawCentredString(18*u, 4*u, text)
    # now draw the tip
    penciltip(canvas,debug=0)
    # draw broken lines across the body.
    canvas.setDash([10,5,16,10],0)
    canvas.line(11*u,2.5*u,22*u,2.5*u)
    canvas.line(22*u,7.5*u,12*u,7.5*u)
.. @+node:ekr.20161117164321.395: *7* penciltip
def penciltip(canvas, debug=1):
    from reportlab.lib.colors import tan, black, green
    from reportlab.lib.units import inch
    u = inch/10.0
    canvas.setLineWidth(4)
    if debug:
        canvas.scale(2.8,2.8) # make it big
        canvas.setLineWidth(1) # small lines
    canvas.setStrokeColor(black)
    canvas.setFillColor(tan)
    p = canvas.beginPath()
    p.moveTo(10*u,0)
    p.lineTo(0,5*u)
    p.lineTo(10*u,10*u)
    p.curveTo(11.5*u,10*u, 11.5*u,7.5*u, 10*u,7.5*u)
    p.curveTo(12*u,7.5*u, 11*u,2.5*u, 9.7*u,2.5*u)
    p.curveTo(10.5*u,2.5*u, 11*u,0, 10*u,0)
    canvas.drawPath(p, stroke=1, fill=1)
    canvas.setFillColor(black)
    p = canvas.beginPath()
    p.moveTo(0,5*u)
    p.lineTo(4*u,3*u)
    p.lineTo(5*u,4.5*u)
    p.lineTo(3*u,6.5*u)
    canvas.drawPath(p, stroke=1, fill=1)
    if debug:
        canvas.setStrokeColor(green) # put in a frame of reference
        canvas.grid([0,5*u,10*u,15*u], [0,5*u,10*u])
.. @+node:ekr.20161117164321.396: *6* Standalone imports
.. @+node:ekr.20161117164321.397: *7* test that of standalone imports of leo files
import glob,sys,traceback

def printModules():
    mods = sys.modules.keys()
    mods.sort()
    for mod in mods: g.pr(mod)

def leoModules():
    files = glob.glob(r'%s\*.py' % g.app.loadDir)
    modules = []
    for file in files:
        path,file = g.os_path_split(file)
        module,ext = g.os_path_splitext(file)
        if g.match(module,0,'leo'):
            modules.append(module)
    return modules

def delLeoModules():
    for module in leoModules():
        if module in sys.modules:
            del sys.modules[module]

def test():
    for module in leoModules():
        g.pr(module)
        exec 'import %s' % module in {},{}
        del sys.modules[module]

delLeoModules()        
test()
# printModules()
.. @+node:ekr.20161117164321.398: *7* Script to run in Idle
def test():
    '''Tests whether all files can be imported.'''
    import glob, os, sys, traceback
    dir = r'c:\prog\leoCVS\leo\src'
    files = glob.glob(r'%s\*.py' % dir)
    modules = []
    for file in files:
        path,file = os.path.split(file)
        module,ext = os.path.splitext(file)
        if module[:3] == 'leo':
            modules.append(module)
    for module in modules:
        g.pr(module)
        try:
            exec 'import %s' % module in {},{}
            del sys.modules[module]
        except:
            traceback.print_exc()

def printModules():
    import sys
    mods = sys.modules.keys()
    mods.sort()
    for mod in mods: g.pr(mod)

.. @+node:ekr.20161117164321.399: *6* test local settings (c.redirect_execute_script_output_to_log_pane)
g.es(c.redirect_execute_script_output_to_log_pane)
g.es(c.config.redirect_execute_script_output_to_log_pane)
g.pr('hello')

#assert c.redirect_execute_script_output_to_log_pane is True
#assert c.config.redirect_execute_script_output_to_log_pane is True
.. @+node:ekr.20161117164321.400: *6* test of 4.3 str_ attributes
.. @+node:ekr.20161117164321.401: *7* set
# Set the attribute.
d = {'str_ekr_attribute': 'abc'}
p.v.t.unknownAttributes = d
.. @+node:ekr.20161117164321.402: *7* get
for p in c.allNodes_iter():
    h = p.h
    if hasattr(p.v.t,'unknownAttributes'):
        d = p.v.t.unknownAttributes
        val = d.get('str_ekr_attribute')
        if val:
           g.es('str_ekr_attribute is: %s' % val)
.. @+node:ekr.20161117164321.403: *6* test of an exception in another module
c.testManager.throwAssertionError()
.. @+node:ekr.20161117164321.404: *6* Test of autocompleter
@language python

# Type a period to autocomplete
leoTest

# Type an open paren to bring up calltip.
c.testManager.findAllAtFileNodes
.. @+node:ekr.20161117164321.405: *6* Test of g.getScript with forcePythonSentinels = False
@language html
.. @+node:ekr.20161117164321.406: *7* g.getScript
g.pr('-'*20)
g.pr(g.getScript(c,p,forcePythonSentinels=False))
.. @+node:ekr.20161117164321.407: *7* html stuff
<body>
@others
</body>
.. @+node:ekr.20161117164321.408: *8* body
This is a body
.. @+node:ekr.20161117164321.409: *6* test of NameError traceback
# Comment

g.pr(z)
.. @+node:ekr.20161117164321.410: *6* Test of os.spawnv calls to c.openWith
table = ('spawnv',None,(
    'os.spawnv',[
    r'c:\vim\vim63\gvim.exe',
    ' --servername LEO ',
    ' --remote-silent ',
    ],
    ".py")),

c.frame.menu.createOpenWithMenuFromTable(table)
.. @+node:ekr.20161117164321.411: *6* Test of redirected scipt with error
# To run this test, set @bool redirect_execute_script_output_to_log_pane = True in the @settings tree.

g.pr('hi')
g.pr(c.config.redirect_execute_script_output_to_log_pane)
g.pr(c.xyzzy)
.. @+node:ekr.20161117164321.412: *6* Test of reportBadChars
s = u"ß"

g.reportBadChars(s,"latin_1")

g.pr(g.toEncodedString(s,"latin_1"))
.. @+node:ekr.20161117164321.413: *6* test of tab_width & tab_width ivars
g.pr(c)
g.pr('use_plugins',c.use_plugins)
g.pr('tab_width',c.tab_width)
g.pr('page_width',c.page_width)
.. @+node:ekr.20161117164321.414: *6* Test of template plugin
.. @+node:ekr.20161117164321.415: *7* A node that uses the template
.. @+node:ekr.20161117164321.416: *6* test of using changes to Go To Line number to handle scripts
# Go To Line number now assumes that selected node is
# the root of a script if there is no ancestor @file node.

@others

# last line
.. @+node:ekr.20161117164321.417: *7* node that throws exception
# We should also be able to use the goto line number command to get to the erroneous line.

a = 1/0 # ZeroDivisionError

b = 2
.. @+node:ekr.20161117164321.418: *6* Test of warnings of conflicting shortcuts
# This problem has been around forever.
g.pr('-' * 40)
# Yes. We *do* want to warn in c.config.exists.
g.pr('exists',g.app.config.exists(c,'showMinibuffer','bool'))
val = c.config.getBool('showMinibuffer')
g.pr('bool:showMinibuffer',val)
val = c.config.getShortcut('showMinibuffer')
g.pr('shortcut:showMinibuffer',val)
.. @+node:ekr.20161117164321.419: *6* test k.registerCommand
k = c.keyHandler

def f (event):
    g.es_print('Hello',color='purple')

def f2 (event):
    g.es_print('Hello2',color='purple')

k.registerCommand('print-hello','Alt-Ctrl-Shift-p',f)
k.registerCommand('print-hello2',None,f2)
.. @+node:ekr.20161117164321.420: *6* Tests of leoGlobals
.. @+node:ekr.20161117164321.421: *7* @test g.rawPrint
g.rawPrint("Test of g.rawPrint")
g.redirectStdout()
g.rawPrint("Test of g.rawPrint")
g.restoreStdout()
.. @+node:ekr.20161117164321.422: *7* test of g.pdb
g.pdb()
.. @+node:ekr.20161117164321.423: *7* Test of g.app.debugSwitch
g.pr(g.app.debugSwitch)

g.app.debugSwitch = 0 # 2: drop into pdb

zerodivide = 1 / 0
.. @+node:ekr.20161117164321.424: *6* Tests of rst3 plugin
@nocolor
@pagewidth 100
@language python
.. @+node:ekr.20161117164321.425: *7* @rst ../doc/ListManagerDocs.html
@language python

@ @rst-options
code_mode = False
show_leo_directives = True
number_code_lines = False
@c

#########################
ListManager Documentation
#########################

:Author: Steven Zatz, Modified by EKR.
:Contact: slzatz@hotmail.com
:Date: $Date: 2008/02/14 14:59:04 $
:Status: This is a "work in progress"
:Revision: $Revision: 1.247 $
:Copyright: Application and documentation use the Python license which is compatible with the GPL. 

This is experimental documentation of a program called ListManager, written in
Python and wxPython using Leo to create both the application code and the
associated reST documentation.

ListManager is an application that allows a group of people working on a joint
project to maintain a common list of todos and related items that have owners,
due dates and associated notes. The application uses mysql as its database for
group use and also uses sqlite for locally resident databases for personal
lists. It works in conjunction with Outlook to allow email messages to be sent
to ListManager for inclusion in lists and uses Outlook to mail messages to
users.

.. contents:: Table of Contents
.. @+node:ekr.20161117164321.426: *8* wxListManager.py
@language python
@color
@others

@ @rst-options
code_mode = True
@c
.. @+node:ekr.20161117164321.427: *9* Initial stuff
@ @rst-markup

Nothing unusual in what follows:  we start with the module imports, setting some global constants including Menu Ids and read the ListManager.ini file.
.. @+node:ekr.20161117164321.428: *10* Module Imports
from wxPython.wx import *
from wxPython.lib.mixins.listctrl import wxListCtrlAutoWidthMixin

import os
import time
import pickle
import socket
import select
import random
import ConfigParser
import threading
import re
import sys

from pywintypes import CreateGuid
from win32com.client import Dispatch
#import win32pdh
import win32api
#from win32com.client import constants #--> just needed two constants...

import MySQLdb
import sqlite
import mx.DateTime

from LMDialogs import CalendarDialog, ModifierDialog, TicklerDialog, MailDialog,LoggerDialog, FinishedDialog, FindDialog, EvalDialog, TreeDialog, StartupDialog
#from wxTreeCtrl import TreeDialog

from printout import PrintTable
.. @+node:ekr.20161117164321.429: *11* @rst-no-head About imports
@nocolor

os
    uses ``os.getcwd``, ``os.path.split``, ``os.chdir``, ``os.path.join``, ``os.path.getmtime``, ``os.startfile``, ``os.environ``

time
    uses ``time.sleep``, ``time.asctime``

pickle
    used to serialize data that is moved from Outlook to ListManager via sockets.  

socket
    as noted above, a socket is opened between Outlook and ListManager to move messages back and forth

select
    ListManager selects on the socket to see if there is a message that has been queued by Outlook

random
    used by the reminder popup to select messages

ConfigParser
    not surprisingly, using ConfiParser to parse the ListManager.ini file.  

threading
    more for fun than absolute necessity, a thread is opened on starting the program that constructs the list of owners for items.  In theory, if the datasize and number of Lists were large enough it could delay the appearance of the GUI and its initial responsiveness if we didn't construct the ownerlist in a thread.  On the other hand, it really let me play with threads and with creating a custom event that signalled the construction of the owner list to the main thread by posting a custom event.

re
    mainly using ``re.sub('[\\/:*"<>|\?]','-',f)`` to make sure that files are constructed only with legal characters.  Also searching the body text of nodes using re because it allows case insensitive searches through ``re.compile(pat, re.I)``.

pywintypes.CreateGuid
    probably should use pure python GUID that is in ASPN cookbook but it was easiest to just use the Windows GUID function.  Thank you Mark Hammond for win32all.

win32com.client.Dispatch
    used when launching Outlook to send email messages

win32api
    using win32api.GetUserName() in case there is no user name in the ini file or no ini file

MySQLdb
    using Andy Dustman's python extension module to connect to mysql back-end.

sqlite
    using  D. Richard Hipp's python extension to connect to local sqlite databases

import mx.DateTime
    using Marc-André Lemburg's mx.DateTime for dealing with datetime stuff in the databases

CalendarDialog, ModifierDialog, TicklerDialog, MailDialog,LoggerDialog, FinishedDialog, FindDialog, EvalDialog, TreeDialog, StartupDialog
   should just import LMDialogs and then access each dialog class by LM.WhateverDialog

printout.PrintTable
    There was an existing wxPython print module for printing from tables that I have modified to print Lists.

*#from win32com.client import constants*
    probably not wise but since the app only needs two constants from this module, just set the directly.  If MSFT decides to change the api, this is not good.
.. @+node:ekr.20161117164321.430: *10* Constants
cwd = os.getcwd()
DIRECTORY = os.path.split(cwd)[0]
os.chdir(DIRECTORY)
del cwd

#Outlook Constants
olMailItem = 0x0
olFlagMarked = 0x2

OFFLINE_ONLY = False #False-> Online only  ; True-> Online and Offline possible; REMOTE_HOST = None -> Offline only

VERSION = '1.02'

@ @rst-markup

The following two global constants are needed to create emails through Outlook via COM::

    olMailItem = 0x0
    olFlagMarked = 0x2

For some reason, it seemed easier to just include them explicitly rather than worrying about generating all the Outlook constants in order to use early binding.  I supppose if MSFT changes the api, that would be a problem.
.. @+node:ekr.20161117164321.431: *10* Menu IDs
@ @rst-markup
Menu Ids -- not much more to say although there should be something to say.
@c

#File Menu-----------------#
idNEWLIST = 1000
idOPENLIST = 1010
idCLOSELIST = 1015
idCLOSEALL = 1017
idSAVEAS = 1020
idDELETELIST = 1025
idPAGESETUP = 1030
idPRINT = 1035
idPRINTPREV = 1040
idMAILLIST = 1045
idOFFLINE = 1048
idEXIT = 1050

#Edit Menu-----------------
idCUT = 1055
idCOPY = 1060
idPASTE = 1065
idDELETEITEMS = 1070
idCOMBINEITEMS = 1075
idFIND = 1080

#Item Menu-------------------
idNEWITEM = 1085
idTOGGLEFINISHED = 1090
idEDITOWNER = 1095
idDUEDATE = 1100
idEDITNOTE = 1105
idMAILITEM =1110

#Diplay Menu---------------------
idSHOWFINISHED = 1115
idSHOWALL = 1120
idREFRESH = 1125
idDISPLAYDATE = 1130

#Tool Menu------------------------
idTICKLERACTIVE = 1135
idSHOWNEXT = 1140
idSYNC = 1145
idARCHIVE = 1150
idEVALUATE = 1155
idTOOLPRINT = 1165
idSENDTO = 1170

#Help Menu-------------------------
idABOUT = 1175
idHELP = 1180


.. @+node:ekr.20161117164321.432: *10* Read Config File
config_file = os.path.join(DIRECTORY, "List Manager.ini")
defaults = dict(pw='python', db='listmanager', ext='txt', local='wxLMDB:sqlite', x='700', y='400')
cp = ConfigParser.ConfigParser(defaults=defaults)
cp.read(config_file) #ConfigParser closes the file

USER = cp.has_option('User','user') and cp.get('User','user') or win32api.GetUserName()

# the following all have default values provided in the constructor
PW = cp.get('User','pw')
DB = cp.get('Database','db')
NOTE_EXT = cp.get('Note','ext')
LOCAL_HOST = cp.get('Hosts','local')
X = cp.getint('Configuration','x')
Y = cp.getint('Configuration','y')

# the folloowing default to None
MAIL_LIST_PATH = cp.has_option('Mail','path') and cp.get('Mail','path') or None
QUICK_LIST = cp.has_option('User','quicklist') and cp.get('User','quicklist') or None

# the following default to False
STARTUP_DIALOG = cp.has_option('User','startup_dialog') and cp.getboolean('User','startup_dialog')
DELETE_LIST = cp.has_option('User','delete_list') and cp.getboolean('User','delete_list')
OUTLOOK = cp.has_option('Mail','outlook') and cp.getboolean('Mail','outlook')

if cp.has_option('Hosts','remote'):
    REMOTE_HOST = cp.get('Hosts','remote')
else:
    REMOTE_HOST = None
    OFFLINE_ONLY = True

# reading it again because of the way defaults are handled
cp = ConfigParser.ConfigParser()
cp.read(config_file) #ConfigParser closes the file

if cp.has_section('Synchronization'):
    SYNC_TABLES = [t[1] for t in cp.items('Synchronization')]
else:
    SYNC_TABLES = ['follow_ups']

.. @+node:ekr.20161117164321.433: *11* @rst-no-head About configuration files
@nocolor

.. sidebar:: A typical *List Manager.ini* file:

    ::

        [Files]
        path0 = wxLMDB:sqlite:mine
        path1 = nycpsszatzsql:mysql:follow_ups

        [Database]
        db = listmanager

        [Note]
        ext = txt

        [Synchronization]
        sync2 = follow_ups
	sync1 = test

        [Hosts]
        remote = nycpsszatzsql:mysql
        local = wxLMDB:sqlite

        [User]
        startup_dialog = true
        user = szatz
        pw = python

        [Mail]
	outlook = true
	path = wxLMDB:sqlite:mail_transfer

        [Configuration]
        y = 642
        x = 975

Application uses the ``ConfigParser`` module ito parse the ini file.  Unfortunately, ``ConfigParser`` doesn't work exactly like I think it should although it has been improved in 2.3.  My main issue is in the handling of default options.  The default options specified through the constructor show up in every section.  For example, if you use the items(*section*) method
then in addition to returning a list of tuples with whatever option/value pairs exist in the section, the list will include all the default option/value pairs, which does not make a whole lot of sense to me.  At the least, there should be a 'nodefaults' argument whose default was *False* but which could be set to *True*.  The following methods should have this option:

- items
- options
- has_option

In any event, because a nodefaults option does not exist, I create the ConfigParser object twice -- once with default options and once without them.  

The application will work fine if there is no ini file. In an effort to save some typing but not be too obscure, many of the options are read such that they default to the correct value either through explicit defaults in the constructor or statements that evaluate to *None* or *False*.

    ``QUICK_LIST = cp.has_option('User','quicklist') and cp.get('User','quicklist') or None``

    ``OUTLOOK = cp.has_option('Mail','outlook') and cp.getboolean('Mail,'outlook')``
.. @+node:ekr.20161117164321.434: *9* class ListManager
class ListManager(wxFrame):
    @others

@ @rst-markup

ListManager is the main class in the application and is a sublass of ``wxFrame``, which is typical for a wxPython application.  From a GUI standpoint, the main child window of the ListManager object is a ``wxNoteBook`` object that holds one ``wxListCtrl`` per notebook page and one ``wxListBox``.  The ``wxListCtrl``\s display item information (e.g., name of the item, owners of the item, etc.) for a particular List and the ``wxListBox``\es displays a list of owners that is used to filter the items displayed by the ``wxListCtrl`` object.

Each ``wxListCtrl`` object has its own set of events that it is hooked to (see CreateNewNotebookPage`<< ListControl Events >>`_.
.. @+node:ekr.20161117164321.435: *10* Instantiation
.. @+node:ekr.20161117164321.436: *11* def __init__
def __init__(self, parent, id, title, size):
    wxFrame.__init__(self, parent, id, title, size = size)

    self.SetIcon(wxIcon('bitmaps//wxpdemo.ico', wxBITMAP_TYPE_ICO))
    self.CreateStatusBar()

    << ListManager Attributes >>
    << Menu Setup >>
    << Toolbar Setup >>
    << Menu/Toolbar Events >>
    << Create Controls>>
    << Layout Stuff >>
    << Other Events >>
    << GUI Instance Objects >>
    << Create Socket >>
    << Load Recent Files >>
    << Idle Timer >>

    ownerthread = threading.Thread(target=self.createownerlist)
    ownerthread.start()
    self.ModifierDialog = None

.. @+node:ekr.20161117164321.437: *12* @rst-no-head About the ctor
@nocolor

The ListManager ``__init__`` method is pretty straightforward.  The ``__init__`` arguments are the ones that need to be passed to ``wxFrame __init__`` method. The wxFrame class has the following form:

    ``wxFrame(parent, id, title, pos=wxDefaultPosition, size=wxDefaultSize, style=wxDEFAULT_FRAME_STYLE, name="frame")``

The default style (``wxDEFAULT_FRAME_STYLE``) includes ``wxMINIMIZE_BOX``, ``wxMAXIMIZE_BOX``, ``wxRESIZE_BORDER``, ``wxSYSTEM_MENU``, ``wxCAPTION`` (the latter is the text that appears in the title bar).

``SetIcon`` is a method of ``wxFrame`` that sets the icon in the upper left of the title bar of the frame.  The wxIcon class has the following form:

    ``wxIcon(filename, type, desiredWidth=-1, desiredHeight=-1)``

``CreateStatusBar`` is a method of ``wxFrame``. The wxPython form is:

        ``CreateStatusBar(number=1, style=0, id=-1)``

*number* -->
    number of fields to create. Specify a value greater than 1 to create a multi-field status bar.

``CreateStatusBar`` needs to be called before << Load Recent Files >>.

The various sections of ``__init__`` are explained in their corresponding section::

    << ListManager Attributes >>
    << Menu Setup >>
    << Toolbar Setup >>
    << Menu/Toolbar Events >>
    << Create Controls>>
    << Layout Stuff >>
    << Other Events >>
    << GUI Instance Objects >>
    << Create Socket >>
    << Load Recent Files >>
.. @+node:ekr.20161117164321.438: *12* << List Manager Attributes >>
self.PropertyDicts = []
self.ItemLists = []
self.ListCtrls = []
self.OwnerLBoxes = []

self.L = -1
self.curIdx = -1

self.printdata = wxPrintData()
self.printdata.SetPaperId(wxPAPER_LETTER)
self.printdata.SetOrientation(wxPORTRAIT)

#self._options = {} #would be used in loadconfig

self.copyitems = []    
self.modified = {}
self.tickler_active = False

#there is a wxPanel in the AddListControl method so each wxListCtrl has a different panel as parent
#there is a nb_sizer = wxNotebookSizer(nb) class but doesn't seem to make any difference

self.editor = []

self.Cursors = {}
self.sqlite_connections = []
self.popupvisible = False
self.in_place_editor = None
self.showrecentcompleted = 0

self.LC_font = wxFont(9, wxSWISS, wxNORMAL, wxNORMAL)

self.date_titles = {'createdate':"Create Date",'duedate':"Due Date",'timestamp':"Last Modified",'finisheddate':"Completion Date"}
self.attr2col_num = {'priority':0, 'name':1,'owners':2, 'date':3}

self.FindDialog = FindDialog(self, "Find...", "")
self.EvalDialog = EvalDialog(self, "Evaluate...", "")
.. @+node:ekr.20161117164321.439: *13* @rst
@nocolor

self.PropertyDicts
    list of dictionaries that describe properties of each ListManager List (note that when referring to a collection of ListManager items a capital *L* List and table are used interchangeably).

self.ItemLists
    list of lists that consist of instance objects of class ``Item``.  Each of the lists contained in self.ItemLists correspond to the items that are being displayed in the ListCtrl.  So ``self.Itemlist[2]`` corresponds to the 2nd tab of the notebook and to the items in self.ListCtrls[2].

The class ``Item`` is just an empty class being used as a convenience to hold item attributes::

    class Item:
        pass

The purpose of the class is just to create an object that can have various attributes as follows:

+-----------------+----------------------------------------------------+
|item.id          |GUID                                                |
+-----------------+----------------------------------------------------+
|item.name        |string that describes the item                      |
+-----------------+----------------------------------------------------+
|item.priority    |integer ranging from 1 (high) to 3 (low)            |
+-----------------+----------------------------------------------------+
|item.owners      |list of the form ["Zatz, Steve", "Hoffman, Steve"]  |
+-----------------+----------------------------------------------------+
|item.note        |string that provides additional info on item        |
+-----------------+----------------------------------------------------+
|item.timestamp   |timestamp indicating when an item was last modified |
+-----------------+----------------------------------------------------+
|item.duedate     |default is None; mx.DateTime date                   |
+-----------------+----------------------------------------------------+
|item.createdate  |mx.DateTime.now() mx.DateTime timestamp             |
+-----------------+----------------------------------------------------+
|item.finisheddate|efaut is None; mx.DateTime date                     |
+-----------------+----------------------------------------------------+

self.ListCtrls
    list of of instance objects of class ListCtrls, which are a subclass of wxPython class wxListCtrl.

self.OwnerLBoxes
    list of of instance objects of wxPython class wxListBox, which is a simple one column List Control.

The wxPython constructor for a wxListBox is:

    ``wxListBox(parent, id, pos=wxDefaultPosition, size=wxDefaultSize, choices=[], style=0)``

self.L
    index of the currently active notebook tab.  If there are any tabs in the notebook then one of them is always selected.  If there are no tabs then this is indicated by setting ``self.L = -1``.

self.curIdx
    currently selected row in the active ``ListCtrl``.  There are times like after a row is deleted in which there may be rows visible but no row is selected.

The following lines set the default printer data::

    self.printdata = wxPrintData()
    self.printdata.SetPaperId(wxPAPER_LETTER)
    self.printdata.SetOrientation(wxPORTRAIT)


The wxPython class ``wxPrintData`` holds a variety of information related to printers and printer device contexts. This class is used to create a wxPrinterDC and a wxPostScriptDC. It is also used as a data member of wxPrintDialogData and wxPageSetupDialogData, as part of the mechanism for transferring data between the print dialogs and the application.

self.copyitems
    list that contains item instance objects that have been copied from one list to be moved to another list.

self.modified
    dictionary that contains the information concerning whether any of several elements have been changed.  Chose a dictionary more to test the idea that I could create a simple method that would update the dictionary and here is an example:

    ``EVT_TEXT(self, self.name.GetId(), lambda e: self.modified.update({'name':1}))``

So this lambda function means that if an ``EVT_TEXT`` event occurs then update the dictionary by adding the key to the dictionary (the value is not used and arbitrarily set to 1).  The wxPython form for the macro ``EVT_TEXT`` is:

    ``EVT_TEXT(window, id, func)``

A ``wxEVT_COMMAND_TEXT_UPDATED`` event is generated when the text in a ``wxTextCtrl`` changes and that is what ``EVT_TEXT`` catches. Note that this event will always be sent when the text control’s content changes - whether this is due to user input or comes programmatically (for example, if ``SetValue()`` is called)

self.Cursors
    dictionary that holds the database cursor objects.  For example, it will look like:  ``{'sqlite':<sqlite cursor object>,'nycpsltszatz':<mysql cursor object>}``

self.tickler_active
    booean determines whether the tickler capabililty is active; can be shut off by unchecking Tickler menu item

self.editor
    list that holds the dictionaries that describe the notes that are edited by the external text editor::

        [
        {
        'table': 'mine',
        'host': 'wxLMDB:sqlite',
        'path': 'C:\\DOCUME~1\\STEVEN~1\\LOCALS~1\\Temp\\Journal Scan schedule.txt',
        'id': '1AB34FB9-9EE6-4AFC-8AF0-FFCA50103BF3',
        'time': 1070850894
        }, 
        {
        'table': 'factoids',
        'host': 'wxLMDB:sqlite',
        'path': 'C:\\DOCUME~1\\STEVEN~1\\LOCALS~1\\Temp\\How many cme programs are sponsored- - 91%.txt', 
        'id': '9CAC4D18-DE1C-4535-B9A5-4CDB1AD3F304', 
        'time': 1070850908
        }
        ]

The method that uses self.editor is `<< Check if Edited File has Changed >>`_.

There is a ``wxPanel`` in the ``AddListControl`` method so each ``wxListCtrl`` has a different panel as parent.

There is a nb_sizer = wxNotebookSizer(nb) class but doesn't seem to make any difference.

self.sqlite_connections
    Here because the sqlite connection has a weakreference that deletes it when you want it around

self.popupvisible
    boolean that is used to ensure that two reminder popups aren't visible at the same time.

self.in_place_editor 
    boolean that indicates whether the inplace item name text editor is active or not.

self.showrecentcompleted
    integer that determines the number of days in the past to retain completed items in the display.

self.LC_font
    default font for all of the ``ListCtrls``:  ``self.LC_font = wxFont(9, wxSWISS, wxNORMAL, wxNORMAL)``

The wxPython ``wxFont`` constructor is:

    ``wxFont(pointSize, family, style, weight, underline=False, faceName="", wencoding=wxFONTENCODING_DEFAULT)``

self.date_titles
    dictionary that holds the various dates that are associated with each item and which can be displayed in the date column.  The dictionary is not modified.  We use one column of each ``ListCtrl`` to display any one of the four dates that that the application tracks. This dictionary associates the item attribute with the text that will be displayed in both the column header for the date and in the dropdown that allows you to change the date:  ``self.date_titles = {'createdate':"Create Date",'duedate':"Due Date",'timestamp':"Last Modified",'finisheddate':"Completion Date"}``

self.attr2col_num
    dictionary that associates the item attribute with the column that attribute is displayed in in the ``ListCtrl``:  ``self.attr2col_num = {'priority':0, 'name':1,'owners':2, 'date':3}``

The following lines construct the Find Dialog and the Dialog that catches errors and shows expressions for debugging::

    self.FindDialog = FindDialog(self, "Find...", "")
    self.EvalDialog = EvalDialog(self, "Evaluate...", "")
.. @+node:ekr.20161117164321.440: *12* << Menu Setup >>
filemenu = wxMenu()
filemenu.Append(idNEWLIST, "New List...", "Create a new List")
filemenu.Append(idOPENLIST, "Open List...", "Open a List")
filemenu.Append(idCLOSELIST, "Close", "Close the current List")
filemenu.Append(idCLOSEALL, "Close All", "Close all open Lists")
filemenu.Append(idSAVEAS, "Save As Text File...", "Save the current List")
filemenu.AppendSeparator()
filemenu.Append(idDELETELIST, "Delete List...", "Select a list to delete")
filemenu.AppendSeparator()
filemenu.Append(idPAGESETUP, "Page Setup...")
filemenu.Append(idPRINT, "Print...", "Print the current view")
filemenu.Append(idPRINTPREV, "Print Preview")
filemenu.AppendSeparator()
filemenu.Append(idMAILLIST, "Mail...", "Mail the current view")
filemenu.AppendSeparator()
filemenu.AppendCheckItem(idOFFLINE, "Work Offline")
filemenu.AppendSeparator()
filemenu.Append(idEXIT, "Exit", "Exit the program")

editmenu = wxMenu()
editmenu.Append(idCUT, "Cut\tCtrl+X")
editmenu.Append(idCOPY, "Copy\tCtrl+C")
editmenu.Append(idPASTE, "Paste\tCtrl+V")
editmenu.AppendSeparator()
editmenu.Append(idDELETEITEMS, "Delete")
editmenu.AppendSeparator()
editmenu.Append(idCOMBINEITEMS, "Combine Items...")
editmenu.AppendSeparator()
editmenu.Append(idFIND, "Find...")

itemmenu = wxMenu()
itemmenu.Append(idNEWITEM, "New Item")
itemmenu.AppendSeparator()
itemmenu.Append(idTOGGLEFINISHED, "Toggle Finished")
itemmenu.Append(idEDITOWNER, "Owner...")
itemmenu.Append(idDUEDATE, "Due Date...")
itemmenu.Append(idEDITNOTE, "Note...")
itemmenu.AppendSeparator()
itemmenu.Append(idMAILITEM, "Mail...")

displaymenu = wxMenu()
displaymenu.Append(idSHOWFINISHED, "Show/Hide Finished...")
displaymenu.AppendSeparator()
displaymenu.Append(idSHOWALL, "Show All", "Show all items in the current list")
displaymenu.AppendSeparator()
displaymenu.Append(idREFRESH, "Refresh Display", "Refresh the Display")
displaymenu.Append(idDISPLAYDATE, "Select Date to Display")

toolmenu = wxMenu()
toolmenu.AppendCheckItem(idTICKLERACTIVE, "Tickler Active")
toolmenu.Check(idTICKLERACTIVE,False)
toolmenu.Append(idSHOWNEXT, "Show Next Reminder")
toolmenu.Append(idSYNC, "Synchronize local and remote DBs")
toolmenu.Append(idARCHIVE, "Archive completed items in list...")
toolmenu.Append(idEVALUATE, "Evaluate an expression...")

helpmenu = wxMenu()
helpmenu.Append(idABOUT, "About ListManager")
helpmenu.Append(idHELP, "Help")

menubar = wxMenuBar()
menubar.Append(filemenu, '&File')
menubar.Append(editmenu, 'Edit')
menubar.Append(itemmenu, 'Item')
menubar.Append(displaymenu, 'Display')
menubar.Append(toolmenu, 'Tools')
menubar.Append(helpmenu, 'Help')
self.SetMenuBar(menubar)
toolmenu.Enable(idSHOWNEXT,self.tickler_active)
filemenu.Enable(idDELETELIST,DELETE_LIST)
filemenu.Check(idOFFLINE,OFFLINE_ONLY)

#file history
self.filehistory = wxFileHistory()
self.filehistory.UseMenu(filemenu)

.. @+node:ekr.20161117164321.441: *13* @rst
@nocolor

+------------------------+------------------------------------------------+
|**File Menu**           |                                                |
+------------------------+------------------------------------------------+
| "New List... "         ||nl| ``self.OnNewList``                         |
+------------------------+------------------------------------------------+
| "Open List..."         ||ol| ``self.OnOpenList``                        |
+------------------------+------------------------------------------------+
| "Close"                |``self.OnCloseList``                            |
+------------------------+------------------------------------------------+
| "Close All"            |``self.OnCloseAll``                             |
+------------------------+------------------------------------------------+
| "Save As Text File..." |``self.OnSaveAsText``                           |
+------------------------+------------------------------------------------+
| "Delete List..."       ||de| ``self.OnDeleteList``                      |
+------------------------+------------------------------------------------+
| "Page Setup..."        ||ps| ``self.OnPageSetup``                       |
+------------------------+------------------------------------------------+
| "Print..."             ||pt| ``self.OnPrint``                           |
+------------------------+------------------------------------------------+
| "Print Preview"        ||pp| ``lambda e: self.OnPrint(e, prev=True)``   |
+------------------------+------------------------------------------------+
| "Mail..."              |``self.OnMailView``                             |
+------------------------+------------------------------------------------+
| "Work Offline"         |``self.OnWorkOffline``                          |
+------------------------+------------------------------------------------+
| "Exit"                 |``self.OnExit``                                 |
+------------------------+------------------------------------------------+
| **Edit Menu**          |                                                |
+------------------------+------------------------------------------------+
| "Cut" [Ctrl+X ]        ||ec| ``lambda e: self.OnCopyItems(e, cut=True)``|
+------------------------+------------------------------------------------+
| "Copy" [Ctrl+C]        ||ey| ``self.OnCopyItems``                       |
+------------------------+------------------------------------------------+
| "Paste" [Ctrl+V]       ||ep| ``self.OnPasteItems``                      |
+------------------------+------------------------------------------------+
| "Delete"               ||de| ``self.OnDeleteItems``                     |
+------------------------+------------------------------------------------+
| "Combine Items..."     |``self.OnCombineItems``                         |
+------------------------+------------------------------------------------+
| "Find..."              ||fi| ``self.OnFind``                            |
+------------------------+------------------------------------------------+
| **Item Menu**          |                                                |
+------------------------+------------------------------------------------+
| "New Item"             ||ni| ``self.OnNewItem``                         |
+------------------------+------------------------------------------------+
| "Toggle Finished"      ||co| ``self.OnToggleFinished``                  |
+------------------------+------------------------------------------------+
| "Owner..."             ||ow| ``self.OnEditOwner``                       |
+------------------------+------------------------------------------------+
| "Due Date..."          ||dd| ``self.OnDueDate``                         |
+------------------------+------------------------------------------------+
| "Note..."              ||en| ``self.OnEditNote``                        |
+------------------------+------------------------------------------------+
| "Mail..."              ||mi| ``self.OnMailItem``                        |
+------------------------+------------------------------------------------+
| **Display Menu**       |                                                |
+------------------------+------------------------------------------------+
| "Show/Hide Finished..."|``self.OnShowFinished``                         |
+------------------------+------------------------------------------------+
| "Show All"             |``self.OnShowAll``                              |
+------------------------+------------------------------------------------+
| "Refresh Display"      ||re| ``self.OnRefresh``                         |
+------------------------+------------------------------------------------+
|"Select Date to Display"|``self.OnDisplayDateCategory``                  |
+------------------------+------------------------------------------------+
| **Tool Menu**          |                                                |
+------------------------+------------------------------------------------+
| "Tickler Active"       |``self.OnActivateTickler``                      |
+------------------------+------------------------------------------------+
| "Show Next Reminder"   |``self.OnShowTickler``                          |
+------------------------+------------------------------------------------+
| "Synchronize ..."      |``self.OnSync``                                 |
+------------------------+------------------------------------------------+
| "Archive completed..." |``self.OnArchive``                              |
+------------------------+------------------------------------------------+
| "Evaluate expression"  |``self.OnShowEvaluate``                         |
+------------------------+------------------------------------------------+
| **Help Menu**          |                                                |
+------------------------+------------------------------------------------+
| "About ListManager"    |``self.OnShowAbout``                            |
+------------------------+------------------------------------------------+
| "Help"                 |``self.OnShowHelp``                             |
+------------------------+------------------------------------------------+




.. @+node:ekr.20161117164321.442: *12* << Toolbar Setup >>
tb = self.CreateToolBar(wxTB_HORIZONTAL|wxTB_FLAT)

tb.AddLabelTool(idNEWLIST, "New (local) List", wxBitmap('bitmaps\\new.bmp'), shortHelp="Create New List")
tb.AddLabelTool(idOPENLIST, "Open", wxBitmap('bitmaps\\open.bmp'), shortHelp="Open List")
tb.AddSeparator()
tb.AddLabelTool(idTOOLPRINT, "Print", wxBitmap('bitmaps\\print.bmp'), shortHelp="Print List")
tb.AddLabelTool(idPRINTPREV, "Preview", wxBitmap('bitmaps\\preview.bmp'), shortHelp="Print Preview")
tb.AddLabelTool(idPAGESETUP, "Setup", wxBitmap('bitmaps\\setup.bmp'), shortHelp="Page Setup")
tb.AddSeparator()
tb.AddLabelTool(idNEWITEM, "New Item", wxBitmap('bitmaps\\new_item.bmp'), shortHelp="Create New Item")
tb.AddSeparator()
tb.AddLabelTool(idREFRESH, "Refresh", wxBitmap('bitmaps\\refresh.bmp'), shortHelp="Refresh Display")     
tb.AddSeparator()
tb.AddLabelTool(idEDITNOTE, "Edit Note", wxBitmap('bitmaps\\edit_doc.bmp'), shortHelp="Edit Note")
tb.AddSeparator()
tb.AddLabelTool(idFIND, "Find", wxBitmap('bitmaps\\find.bmp'), shortHelp = "Find Item")        
tb.AddSeparator()
tb.AddLabelTool(idCUT, "Cut", wxBitmap('bitmaps\\editcut.bmp'), shortHelp ="Cut Item")        
tb.AddLabelTool(idCOPY, "Copy", wxBitmap('bitmaps\\copy.bmp'), shortHelp ="Copy Item")
tb.AddLabelTool(idPASTE, "Paste", wxBitmap('bitmaps\\paste.bmp'), shortHelp="Paste Item")
tb.AddSeparator()
tb.AddLabelTool(idTOGGLEFINISHED, "Toggle Date", wxBitmap('bitmaps\\filledbox.bmp'), shortHelp="Toggle Finished Date")
tb.AddLabelTool(idDELETEITEMS, "Delete", wxBitmap('bitmaps\\delete.bmp'), shortHelp="Delete Item")
tb.AddLabelTool(idDUEDATE, "Due Date", wxBitmap('bitmaps\\calendar.bmp'), shortHelp="Enter Due Date")
tb.AddLabelTool(idEDITOWNER,"Owner", wxBitmap('bitmaps\\owners.bmp'), shortHelp="Select Owner(s)")
tb.AddSeparator()
tb.AddLabelTool(idMAILITEM, "Mail", wxBitmap('bitmaps\\mail.bmp'), shortHelp="Mail Item")

if QUICK_LIST:
    tb.AddSeparator()
    tb.AddLabelTool(idSENDTO, "Send to", wxBitmap('bitmaps\\sendto.bmp'), shortHelp="Send to %s"%QUICK_LIST)

tb.Realize()
.. @+node:ekr.20161117164321.443: *12* << Menu/Toolbar Events >>
#File Menu ------------------------------------
EVT_MENU(self, idNEWLIST, self.OnNewList)
EVT_MENU(self, idOPENLIST, self.OnOpenList)
EVT_MENU(self, idCLOSELIST, self.OnCloseList)
EVT_MENU(self, idCLOSEALL, self.OnCloseAll)
EVT_MENU(self, idSAVEAS, self.OnSaveAsText)
EVT_MENU(self, idDELETELIST, self.OnDeleteList)
EVT_MENU(self, idPAGESETUP, self.OnPageSetup)
EVT_MENU(self, idPRINT, self.OnPrint)
EVT_MENU(self, idPRINTPREV, lambda e: self.OnPrint(e, prev=True))
EVT_MENU(self, idOFFLINE, self.OnWorkOffline)
EVT_MENU(self, idMAILLIST, self.OnMailView)      
EVT_MENU_RANGE(self, wxID_FILE1, wxID_FILE9, self.OnFileList)
EVT_MENU(self, idEXIT, self.OnExit)
#Edit Menu ------------------------------------
EVT_MENU(self, idCUT, lambda e: self.OnCopyItems(e, cut=True))        
EVT_MENU(self, idCOPY, self.OnCopyItems)
EVT_MENU(self, idPASTE, self.OnPasteItems)
EVT_MENU(self, idDELETEITEMS, self.OnDeleteItems)
EVT_MENU(self, idCOMBINEITEMS, self.OnCombineItems)
EVT_MENU(self, idFIND, self.OnFind)
#item Menu ------------------------------------
EVT_MENU(self, idNEWITEM, self.OnNewItem)
EVT_MENU(self, idTOGGLEFINISHED, self.OnToggleFinished)             
EVT_MENU(self, idDUEDATE, self.OnDueDate)
EVT_MENU(self, idEDITOWNER, self.OnEditOwner)
EVT_MENU(self, idEDITNOTE, self.OnEditNote)
EVT_MENU(self, idMAILITEM, self.OnMailItem)
#Dips Menu ------------------------------------
EVT_MENU(self, idSHOWFINISHED, self.OnShowFinished)
EVT_MENU(self, idSHOWALL, self.OnShowAll)
EVT_MENU(self, idREFRESH, self.OnRefresh)
EVT_MENU(self, idDISPLAYDATE, self.OnDisplayDateCategory)
#Tool Menu ---------------------------------------
EVT_MENU(self, idTICKLERACTIVE, self.OnActivateTickler)
EVT_MENU(self, idSHOWNEXT, self.OnShowTickler)
EVT_MENU(self, idSYNC, self.OnSync)
EVT_MENU(self, idARCHIVE, self.OnArchive)
EVT_MENU(self, idEVALUATE, self.OnShowEvaluate)
#Help Menu -----------------------------------------
EVT_MENU(self, idABOUT, self.OnShowAbout)
EVT_MENU(self, idHELP, self.OnShowHelp)

EVT_TOOL(self, idTOOLPRINT, lambda e: self.OnPrint(e,showprtdlg=False))

if QUICK_LIST:
    EVT_TOOL(self, idSENDTO, lambda e: self.OnMoveToSpecificList(e,QUICK_LIST))
.. @+node:ekr.20161117164321.444: *12* << Create Controls>>
upper_panel = wxPanel(self, -1)   #size = (900,400)
bottom_panel = wxPanel(self, -1, size = (900,150)) #900 note that 000 seems to work???

nb = wxNotebook(upper_panel, -1, size=(900,500), style=wxNB_BOTTOM)

f = wxFont(10, wxSWISS, wxNORMAL, wxNORMAL)
self.name = wxTextCtrl(bottom_panel, -1, size = (285,42), style = wxTE_MULTILINE|wxTE_RICH2)#34 #wxTE_PROCESS_ENTER
self.name.SetDefaultStyle(wxTextAttr("BLACK", font = f))

self.owners = wxTextCtrl(bottom_panel, -1, size = (250,42),style = wxTE_MULTILINE|wxTE_RICH2)
self.owners.SetDefaultStyle(wxTextAttr("BLACK", font = f))

self.note = wxTextCtrl(bottom_panel, -1, size = (400,50), style=wxTE_MULTILINE)

.. @+node:ekr.20161117164321.445: *12* << Other Events >>
EVT_TEXT(self, self.name.GetId(), lambda e: self.modified.update({'name':1}))
EVT_TEXT(self, self.note.GetId(), lambda e: self.modified.update({'note':1}))
EVT_TEXT(self, self.owners.GetId(), lambda e: self.modified.update({'owners':1}))

EVT_CLOSE(self, self.OnWindowExit)

EVT_IDLE(self, self.OnIdle)

.. @+node:ekr.20161117164321.446: *12* << Layout Stuff >>
#Appears necessary to really get the listcontrol to size with the overall window  
#upper_panel sizer
sizer = wxBoxSizer(wxHORIZONTAL)
sizer.Add(nb,1,wxALIGN_LEFT|wxEXPAND)
upper_panel.SetSizer(sizer)        

#sizer for the row of data items
box = wxBoxSizer(wxHORIZONTAL)
box.Add(self.name,1,wxEXPAND)
box.Add(self.owners,0)

#bottom_panel sizer  
sizer = wxBoxSizer(wxVERTICAL)        
sizer.AddSizer(box, 0, wxGROW|wxALIGN_CENTER_VERTICAL|wxALL, 5)
sizer.Add(self.note,1,wxALIGN_LEFT|wxEXPAND)
bottom_panel.SetSizer(sizer)

sizer = wxBoxSizer(wxVERTICAL)
sizer.Add(upper_panel,1,wxALIGN_TOP|wxEXPAND)
sizer.Add(bottom_panel,0,wxALIGN_TOP|wxEXPAND)

self.SetAutoLayout(1)
self.SetSizer(sizer)
#sizer.Fit(self) #actively does bad things to the dimensions on startup
.. @+node:ekr.20161117164321.447: *12* << GUI Instance Objects >>
self.toolmenu = toolmenu
self.filemenu = filemenu
self.nb = nb
self.tb = tb
.. @+node:ekr.20161117164321.448: *12* << Create Socket >>
if OUTLOOK:
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # Create a TCP socket
    s.bind(('localhost',8888)) # Bind to port 8888
    s.listen(5) # Listen, but allow no more than
    self.sock = s
.. @+node:ekr.20161117164321.449: *12* << Load Recent Files >>
try:
    pathlist = [f[1] for f in cp.items('Files')]
except:
    pathlist = []

if pathlist:
    pathlist.sort()
    pathlist.reverse()
    for path in pathlist[1:]:
        self.OnFileList(path=path)

    #don't want to trigger the page change event until n-1 of n files are loaded
    EVT_NOTEBOOK_PAGE_CHANGED(self,nb.GetId(),self.OnPageChange)

    self.OnFileList(path=pathlist[0])
else:
    EVT_NOTEBOOK_PAGE_CHANGED(self,nb.GetId(),self.OnPageChange)



.. @+node:ekr.20161117164321.450: *12* << Idle Timer >>
ID_TIMER = wxNewId()
self.timer = wxTimer(self, ID_TIMER) 
EVT_TIMER(self,  ID_TIMER, self.OnIdle)
self.timer.Start(3000)
.. @+node:ekr.20161117164321.451: *10* Ownerlist creation methods (used by thread)
.. @+node:ekr.20161117164321.452: *11* def createownerlist
def createownerlist(self):

    if REMOTE_HOST and OFFLINE_ONLY is False:
        cursor = self.GetCursor(REMOTE_HOST)
        sql = "SHOW TABLES" #sorted
    else:
        cursor = self.GetCursor(LOCAL_HOST)
        sql = "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"

    cursor.execute(sql)
    results = cursor.fetchall()

    #excluding 'system' tables and archive tables
    excluded_tables = ['user_sync','sync','owners']
    tables = [t for (t,) in results if t.find('_archive')== -1 and t not in excluded_tables]

    sql_list = []
    for table in tables:
        sql_list.append("""SELECT owner1 FROM %s UNION SELECT owner2 FROM %s UNION SELECT owner3 FROM %s"""%((table,)*3))

    sql = " UNION ".join(sql_list)
    cursor.execute(sql)
    results = cursor.fetchall()

    _list = [x[0] for x in results]
    if '' in _list:
        _list.remove('')
    if None in _list:
        _list.remove(None)

    self._list = _list

    #posting custom event to signal that this thread is done
    evt = wxPyEvent()
    evt_id = wxNewEventType()
    evt.SetEventType(evt_id)
    self.Connect(-1, -1, evt_id, self.createownerdialog)
    wxPostEvent(self, evt)

.. @+node:ekr.20161117164321.453: *11* def createownerdialog
def createownerdialog(self, evt=None):
    self.ModifierDialog = ModifierDialog(parent=self, title="Select owner(s)", size=(180,300), style=wxCAPTION, modifierlist = self._list)
    del self._list

.. @+node:ekr.20161117164321.454: *10* Notebook methods
.. @+node:ekr.20161117164321.455: *11* def CreateNewNotebookPage
def CreateNewNotebookPage(self, host, table):

    Properties = {'owner':'*ALL',
                'LCdate':'duedate',
                'sort':{'attribute':'priority','direction':0}, #these could be set in Config
                'showfinished':0} #-1 show them all; 0 show none; integer show for that many days

    Properties['table'] = table
    Properties['host'] = host

    self.PropertyDicts.append(Properties)

    self.L = len(self.ItemLists)#could use self.ListCtrls, self.OwnerLBoxes, etc. with a -1

    results = self.ReadFromDB()
    if results is None:
        self.PropertyDicts = self.PropertyDicts[:-1]
        self.L = self.L - 1
        return

    panel = wxPanel(self.nb, -1, size = (900,400))
    LCtrl = ListCtrl(panel, -1, style=wxLC_REPORT|wxSUNKEN_BORDER|wxLC_VRULES|wxLC_HRULES)
    LCtrl.SetFont(self.LC_font)
    self.ListCtrls.append(LCtrl)

    OLBox = wxListBox(panel, -1, size = (126,550), choices = [""], style=wxLB_SORT|wxSUNKEN_BORDER)
    self.OwnerLBoxes.append(OLBox)

    sizer = wxBoxSizer(wxHORIZONTAL)
    sizer.Add(OLBox,0,wxALIGN_LEFT|wxEXPAND)
    sizer.Add(LCtrl,1,wxALIGN_LEFT|wxEXPAND)
    panel.SetSizer(sizer)

    self.ItemLists.append(self.CreateAndDisplayList(results)) 

    << Fill OwnerListBox >>
    << ListControl Events >>

    #img_num = LCtrl.arrows[Properties['sort']['direction']]
    #LCtrl.SetColumnImage(self.attr2col_num[Properties['sort']['attribute']], img_num)

    rdbms = host.split(':')[1]
    if rdbms == 'mysql':
        tab_title = '%s (remote)'%table
    else:
        tab_title = table

    if table in SYNC_TABLES:
        tab_title = '*'+tab_title

    self.nb.AddPage(panel,tab_title)
    self.nb.SetSelection(self.L)

    self.filehistory.AddFileToHistory('%s:%s'%(host,table))

    self.SetStatusText("Successfully loaded %s"%tab_title)

.. @+node:ekr.20161117164321.456: *12* << Fill OwnerListBox >>
cursor = self.GetCursor(host)
if cursor is None:
    g.pr("Couldn't get cursor to fill OwnerListBox")
    return

cursor.execute("SELECT owner1 FROM %s UNION SELECT owner2 FROM %s UNION SELECT owner3 FROM %s"%((table,)*3))

owners = [x for (x,) in cursor.fetchall()]

if None in owners:
    owners.remove(None)
if '' in owners:
    owners.remove('')

OLBox.Clear()
for name in owners: 
    OLBox.Append(name)
OLBox.Append('*ALL')
OLBox.SetSelection(0)

.. @+node:ekr.20161117164321.457: *12* << ListControl Events >>
LCId = LCtrl.GetId()
EVT_LIST_ITEM_SELECTED(self, LCId, self.OnItemSelected)
EVT_LIST_ITEM_ACTIVATED(self, LCId, self.OnDisplayInPlaceEditor)
EVT_LEFT_DOWN(LCtrl, self.OnLeftDown) 
EVT_LEFT_DCLICK(LCtrl, self.OnLeftDown)
EVT_RIGHT_DOWN(LCtrl, self.OnRightDown)
EVT_LIST_COL_CLICK(self, LCId, self.OnColumnClick)
EVT_LIST_COL_RIGHT_CLICK(self, LCId, self.OnColumnRightClick)

# the following is a ListBox event
EVT_LISTBOX(self, OLBox.GetId(), self.OnFilterOwners)

.. @+node:ekr.20161117164321.458: *11* def OnPageChange
def OnPageChange(self, evt=None):
    if self.modified:
        self.OnUpdate()

    self.L = L = self.nb.GetSelection()

    << Find Highlighted Row >>
    << Update Title >>

    evt.Skip() #051403

.. @+node:ekr.20161117164321.459: *12* << Find Highlighted Row >>
idx = self.ListCtrls[L].GetNextItem(-1, wxLIST_NEXT_ALL, wxLIST_STATE_SELECTED)
if idx != -1:
    self.curIdx = idx
    #LCtrl.EnsureVisible(idx)
    self.OnItemSelected()
elif self.ItemLists[L]:
    self.curIdx = 0
    self.ListCtrls[L].SetItemState(0, wxLIST_STATE_SELECTED, wxLIST_STATE_SELECTED)
    #the line above triggers an OnItemSelected EVT so don't need self.OnItemSelected() 092803
else:
    self.curIdx = -1

.. @+node:ekr.20161117164321.460: *12* << Update Title >>
location,rdbms = self.PropertyDicts[L]['host'].split(':')
table = self.PropertyDicts[L]['table']
self.SetTitle("List Manager:  %s:  %s:  %s"%(location,rdbms,table))

.. @+node:ekr.20161117164321.461: *10* Tickler methods
.. @+node:ekr.20161117164321.462: *11* def OnShowTickler
def OnShowTickler(self, evt=None):
    if self.popupvisible:
        return

    self.popupvisible = True

    host = 'wxLMDB:sqlite'
    cursor = self.Cursors[host]
    table = 'follow_ups'

    sql = "SELECT COUNT() FROM "+table+" WHERE finisheddate IS NULL AND priority > 1"
    cursor.execute(sql)
    results = cursor.fetchone()

    num_items = int(results[0])

    if not num_items:
        return

    if self.modified: #Should decide if this should be put back or not
        self.OnUpdate()

    n = random.randint(0,num_items-1)

    sql = "SELECT priority,name,createdate,finisheddate,duedate,owner1,owner2,owner3,id,timestamp,note FROM "+table+" WHERE finisheddate IS NULL AND priority > 1 LIMIT 1 OFFSET %d"%n

    try:
        cursor.execute(sql)
    except:
        g.pr("In OnShowTickler and attempt to Select an item failed")
        return

    row = cursor.fetchone()

    class Item: pass
    item = Item()

    item.priority = int(row[0]) #int(row[0]) needs int because it seems to come back as a long from MySQL
    item.name = row[1]
    item.createdate = row[2]
    item.finisheddate = row[3]
    item.duedate = row[4]
    item.owners = [z for z in row[5:7] if z is not None] #if you carry around ['tom',None,None] you have an issue when you go write it
    item.id = row[8]
    item.timestamp = row[9]
    item.note = row[10]

    dlg = TicklerDialog(self, "", "Do something about this!!!", size=(550,350))
    TC = dlg.TC

    f = wxFont(14, wxSWISS, wxITALIC, wxBOLD, False)
    TC.SetDefaultStyle(wxTextAttr("BLUE",wxNullColour, f))
    TC.AppendText("%s..."%item.name)

    if item.priority == 3:
        TC.SetDefaultStyle(wxTextAttr("RED","YELLOW",f))
    TC.AppendText("%d\n\n"%item.priority)

    f = wxFont(8, wxSWISS, wxNORMAL, wxNORMAL)
    TC.SetDefaultStyle(wxTextAttr("BLACK","WHITE", f))
    TC.AppendText("owners: %s\n"%", ".join(item.owners))
    TC.AppendText("created on: %s\n"%item.createdate.Format('%m/%d/%y'))
    if item.duedate:
        ddate = item.duedate.Format('%m/%d/%y')
    else:
        ddate = "<no due date>"
    TC.AppendText("due on: %s\n\n"%ddate)

    note = item.note
    if not note:
        note = "<no note>"
    TC.AppendText("%s\n\n"%note)
    f = wxFont(10, wxSWISS, wxITALIC, wxBOLD)
    TC.SetDefaultStyle(wxTextAttr("BLACK",wxNullColour, f))
    TC.AppendText('follow_ups')
    TC.ShowPosition(0)   #did not do anything
    TC.SetInsertionPoint(0)
    result = dlg.ShowModal()
    dlg.Destroy()
    self.popupvisible = False     

    if result in (wxID_OK, wxID_APPLY):

        for L,Properties in enumerate(self.PropertyDicts):
            if Properties['table'] == table:
                break
        else:
            g.pr("Can't find %s"%table)
            return

        self.nb.SetSelection(L) #if the page changes it sends a EVT_NOTEBOOK_PAGE_CHANGED, which calls OnPageChange
        self.L = L
        self.FindNode(item)
        if result==wxID_APPLY:
            self.OnMailItem(item)

    elif result==wxID_FORWARD:
        self.OnShowTickler()

.. @+node:ekr.20161117164321.463: *11* def OnActivateTickler
def OnActivateTickler(self, evt):
    self.tickler_active = not self.tickler_active
    self.toolmenu.Enable(idSHOWNEXT,self.tickler_active)


.. @+node:ekr.20161117164321.464: *10* Email methods
.. @+node:ekr.20161117164321.465: *11* OnMailItem
def OnMailItem(self, evt=None, item=None):
    if item is None:
        if self.curIdx == -1:
            return
        else:
            item = self.ItemLists[self.L][self.curIdx]

    dlg = MailDialog(self,"Mail a reminder", size=(450,500),
               recipients=item.owners,    
               subject=item.name,
               body=self.GetNote())          
    result = dlg.ShowModal()
    if result==wxID_OK:
        outlook= Dispatch("Outlook.Application")
        newMsg = outlook.CreateItem(olMailItem) #outlook.CreateItem(constants.olMailItem)
        newMsg.To = to = dlg.RTC.GetValue()
        newMsg.Subject = subject = dlg.STC.GetValue()
        newMsg.Body = body = dlg.BTC.GetValue()

        #newMsg.FlagStatus = constants.olFlagMarked

        newMsg.Display()

        dlg.Destroy()            
        #del outlook

        self.note.SetSelection(0,0)
        self.note.WriteText("**************************************************\n")
        self.note.WriteText("Email sent on %s\n"%mx.DateTime.today().Format("%m/%d/%y"))
        self.note.WriteText("To: %s\n"%to)
        self.note.WriteText("Subject: %s\n"%subject)
        self.note.WriteText("%s\n"%body)
        self.note.WriteText("**************************************************\n")

.. @+node:ekr.20161117164321.466: *11* OnMailView
def OnMailView(self, evt=None):
    recipients = [self.PropertyDicts[self.L]['owner']]

    body = ""
    for i,item in enumerate(self.ItemLists[self.L]):
        body = body+"%d. %s (%d)\n"%(i+1, item.name, item.priority)

    subject = "Follow-ups " + mx.DateTime.today().Format("%m/%d/%y")

    dlg = MailDialog(self,"Follow-up List", size=(450,500),
               recipients=recipients,
               subject=subject,
               body=body)

    val = dlg.ShowModal()
    dlg.Destroy()
    if val==wxID_OK:
        outlook= Dispatch("Outlook.Application")
        newMsg = outlook.CreateItem(olMailItem) #outlook.CreateItem(constants.olMailItem)
        newMsg.To = dlg.RTC.GetValue()
        newMsg.Subject = dlg.STC.GetValue()
        newMsg.Body = dlg.BTC.GetValue()

        newMsg.FlagStatus = olFlagMarked #constants.olFlagMarked
        newMsg.Categories = "Follow-up"

        newMsg.Display()

        #del outlook

.. @+node:ekr.20161117164321.467: *10* Cut/Copy/Paste methods
.. @+node:ekr.20161117164321.468: *11* OnCopyItems
def OnCopyItems(self, event=None, cut=False):
    if self.curIdx == -1:
        return

    L = self.L
    IList = self.ItemLists[L]
    LCtrl = self.ListCtrls[L]

    << Find Highlighted Items >>

    self.SetStatusText("%d items copied"%len(copyitems))
    if cut:
        self.OnDeleteItems()

.. @+node:ekr.20161117164321.469: *12* << Find Highlighted Items >>
copyitems = []
i = -1
while 1:
    i = LCtrl.GetNextItem(i, wxLIST_NEXT_ALL, wxLIST_STATE_SELECTED)
    if i==-1:
        break
    item = IList[i]
    item.notes = self.GetNote(L,item) #handles the database situation
    copyitems.append(item)

self.copyitems = copyitems
.. @+node:ekr.20161117164321.470: *11* OnPasteItems
def OnPasteItems(self, evt=None, L=None): #noselect 051603
    #used by OnMoveToList, OnMoveToSpecificList and called directly
    if not self.copyitems:
        g.pr("Nothing was selected to be copied")
        return

    if L is None: #this is not needed by OnMoveTo or OnDragToTab but is for a straight call
        L = self.L

    Properties = self.PropertyDicts[L]
    LCtrl = self.ListCtrls[L]
    IList = self.ItemLists[L]

    items = self.copyitems
    numitems = len(items)

    for item in items:

        z = item.owners+[None,None,None]

        id = self.GetUID() #we do give it a new id
        host = Properties['host']
        cursor = self.Cursors[host]
        table = Properties['table']

        createdate = mx.DateTime.now() #need this or else it won't be seen as a new item when synching; would be seen as updated
        command = "INSERT INTO "+table+" (priority,name,createdate,finisheddate,duedate,note,owner1,owner2,owner3,id) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)"
        cursor.execute(command,(item.priority,item.name,createdate,item.finisheddate,item.duedate,item.notes,z[0],z[1],z[2],id))

        timestamp = self.TimeStamper(host, cursor, table, id)

        #creating a new item breaks the connection between item.x and new_item.x
        class Item: pass
        new_item = Item()
        new_item.id = id
        new_item.priority = item.priority
        new_item.owners = item.owners
        new_item.name = item.name
        new_item.timestamp = timestamp
        new_item.duedate =item.duedate
        new_item.finisheddate = item.finisheddate
        new_item.createdate = createdate
        IList.insert(0,new_item)

    self.DisplayList(IList,L)

    #If we didn't come from OnMoveToList or OnMoveToSpecificList where L != self.L
    if L==self.L:
        for i in range(numitems):
            LCtrl.SetItemState(i, wxLIST_STATE_SELECTED, wxLIST_STATE_SELECTED)
        self.curIdx = numitems-1



.. @+node:ekr.20161117164321.471: *11* OnDeleteItems
def OnDeleteItems(self, event=None):
    """Called directly and by OnCopyItems (cut = true)
    """
    if self.curIdx == -1: #not absolutely necessary but gets you out quickly
        return

    L = self.L
    LCtrl = self.ListCtrls[L]
    IList = self.ItemLists[L]
    Properties = self.PropertyDicts[L]

    i = -1
    while 1:
        i = LCtrl.GetNextItem(i, wxLIST_NEXT_ALL, wxLIST_STATE_SELECTED)
        if i==-1:
            break
        item = IList.pop(i)
        LCtrl.DeleteItem(i)

        host = Properties['host']
        cursor = self.Cursors[host]
        table = Properties['table']

        cursor.execute("DELETE from "+table+" WHERE id = %s", (item.id,))

        #Track Deletes for Syncing ############################################
        if table in SYNC_TABLES:
            if host.split(':')[1] == 'sqlite':
                timestamp = mx.DateTime.now()
                cursor.execute("INSERT INTO sync (id,action,table_name,name,timestamp) VALUES (%s,%s,%s,%s,%s)",(item.id,'d',table,item.name,timestamp))
            else:
                cursor.execute("INSERT INTO sync (id,action,table_name,user,name) VALUES (%s,%s,%s,%s,%s)",(item.id,'d',table,USER,item.name))
        #########################################################################
        i-=1

    self.name.Clear()
    self.owners.Clear()
    self.note.Clear()
    #note that Clearing does cause self.modified -->{'name':1}
    self.modified = {}
    self.curIdx = -1

.. @+node:ekr.20161117164321.472: *10* MouseDown methods
.. @+node:ekr.20161117164321.473: *11* OnLeftDown (Action depends on x coordinate)
def OnLeftDown(self, evt):
    g.pr("Here")
    if self.modified:
        #if inplace editor is open and you click anywhere (same or different row from current row) but in the editor itself then just to close editor
        flag = self.modified.has_key('inplace')
        self.OnUpdate()
        if flag:
            evt.Skip() #without Skip, EVT_LIST_ITEM_SELECTED is not generated if you click in a new row
            return

    x,y = evt.GetPosition()
    LCtrl = self.ListCtrls[self.L]

    #Using HitTest to obtain row clicked on because there was a noticable delay in the generation of an
    #EVT_LIST_ITEM_SELECTED event when you click on the already selected row
    idx,flags = LCtrl.HitTest((x,y))

    #if you are below rows of items then idx = -1 which could match self.curIdx = -1
    if idx == -1:
        return

    # only if you click on the currently selected row do the following events occur
    if idx == self.curIdx:
        if x < 18:
            self.OnToggleFinished()
        elif x < 33:
            self.OnPriority()
        elif x < 33 + LCtrl.GetColumnWidth(1):
            self.OnDisplayInPlaceEditor()
        elif x < 33 + LCtrl.GetColumnWidth(1) + LCtrl.GetColumnWidth(2): 
            self.OnEditOwner()
        else:
            self.OnDueDate
    else:
        evt.Skip() #without Skip, EVT_LIST_ITEM_SELECTED is not generated if you click in a new row



.. @+node:ekr.20161117164321.474: *11* OnRightDown (Display popup sendto menu)
def OnRightDown(self, evt):
    x,y = evt.GetPosition()

    sendtomenu = wxMenu()

    open_tables = []
    for page,Properties in enumerate(self.PropertyDicts):
        host,table = Properties['host'],Properties['table']
        open_tables.append((host,table))
        sendtomenu.Append(1+page,"%s (%s)"%(table,host))
        EVT_MENU(self, 1+page, lambda e,p=page: self.OnMoveToList(e,p))

    sendtomenu.Delete(self.L+1) # don't send it to the page you're already on
    sendtomenu.AppendSeparator()

    self.closed_tables = []
    for host,cursor in self.Cursors.items():

        location, rdbms = host.split(':')

        if rdbms == 'sqlite':
            cursor.execute("SELECT name FROM sqlite_master WHERE type='table' ORDER BY name")
        elif rdbms == 'mysql':
            cursor.execute("SHOW tables")

        results = cursor.fetchall()

        page+=1
        for (table,) in results:
            if not ((host,table) in open_tables or table in ['user_sync','owners','sync']):
                self.closed_tables.append((host,table))
                sendtomenu.Append(1+page,"%s (%s)"%('*'+table,host))
                EVT_MENU(self, 1+page, lambda e,p=page: self.OnMoveToList(e,p))
                page+=1

    self.PopupMenu(sendtomenu,(x+125,y+40))
    sendtomenu.Destroy()

.. @+node:ekr.20161117164321.475: *10* Move/Combine items methods
.. @+node:ekr.20161117164321.476: *11* OnCombineItems
def OnCombineItems(self, evt):
    L = self.L
    idx = self.curIdx
    IList = self.ItemLists[L]
    LCtrl = self.ListCtrls[L]

    combine_list = []
    i = -1
    while 1:
        i = LCtrl.GetNextItem(i, wxLIST_NEXT_ALL, wxLIST_STATE_SELECTED)
        if i==-1:
            break
        combine_list.append((IList[i].createdate,IList[i]))


    if len(combine_list) < 2:
        g.pr("Fewer than two items highlighted")
        return

    combine_list.sort()
    combine_list.reverse()

    dlg = wxMessageDialog(self,
                        "Combine the %d selected items?"%len(combine_list),
                        "Combine Items?",
                        wxICON_QUESTION|wxYES_NO)

    if dlg.ShowModal() == wxID_YES:
        Properties = self.PropertyDicts[L]
        host = Properties['host']
        cursor = self.Cursors[host]
        table = Properties['table']

        t_item = combine_list[0][1]
        merge_list = combine_list[1:]
        new_note = ""

        for date,item in merge_list:
            note = self.GetNote(item=item)
            date = date.Format("%m/%d/%y")
            new_note = "%s\n%s %s\n\n%s"%(new_note, date, item.name, note)

            cursor.execute("DELETE from "+table+" WHERE id = %s", (item.id,))
            #Track Deletes for Syncing ############################################
            if table in SYNC_TABLES:
                if host.split(':')[1] == 'sqlite':
                    timestamp = mx.DateTime.now()
                    cursor.execute("INSERT INTO sync (id,action,table_name,name,timestamp) VALUES (%s,%s,%s,%s,%s)",(item.id,'d',table,item.name,timestamp))
                else:
                    cursor.execute("INSERT INTO sync (id,action,table_name,user,name) VALUES (%s,%s,%s,%s,%s)",(item.id,'d',table,USER,item.name))
            #########################################################################

        t_note = self.GetNote(item=t_item)
        t_note = "%s\n%s"%(t_note,new_note)

        #What about combining owners?######################################

        cursor.execute("UPDATE "+table+" SET name = %s, note = %s WHERE id = %s", (t_item.name+"*",t_note,t_item.id))
        t_item.timestamp = self.TimeStamper(host, cursor, table, t_item.id)

        self.OnRefresh()
        LCtrl.SetItemState(0, 0, wxLIST_STATE_SELECTED)
        IList = self.ItemLists[L]
        id = t_item.id
        idx = -1
        for item in IList:
            idx+=1
            if id == item.id:
                break
        else:
            idx = -1 

        #should never be -1
        if idx != -1:	
            LCtrl.SetItemState(idx, wxLIST_STATE_SELECTED, wxLIST_STATE_SELECTED)
            LCtrl.EnsureVisible(idx)
        self.curIdx = idx

    dlg.Destroy()

.. @+node:ekr.20161117164321.477: *11* OnMoveToList
def OnMoveToList(self, evt=None, page=0):
    self.OnCopyItems(cut=True)
    pc = self.nb.GetPageCount()
    if page < pc:		
        self.OnPasteItems(L=page)
    else:
        host,table = self.closed_tables[page-pc]
        cursor = self.Cursors[host]# in ini self.Cursors[host]

        for item in self.copyitems:
            z = item.owners+[None,None,None]
            id = self.GetUID() #give it a new id

            #need this or else it won't be seen as a new item when syncing; would be seen as updated
            createdate = mx.DateTime.now() 
            command = "INSERT INTO "+table+" (priority,name,createdate,finisheddate,duedate,note,owner1,owner2,owner3,id) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)"
            cursor.execute(command,(item.priority,item.name,createdate,item.finisheddate,item.duedate,item.notes,z[0],z[1],z[2],id))
            timestamp = self.TimeStamper(host, cursor, table, id)

    self.copyitems = []

.. @+node:ekr.20161117164321.478: *11* OnMoveToSpecificList
def OnMoveToSpecificList(self, evt=None, table='follow_ups'):
    matches = {}
    for page,Properties in enumerate(self.PropertyDicts):
        host,tble = Properties['host'],Properties['table']
        if tble == table:
            rdbms = host.split(':')[1]
            matches[rdbms] = page

    self.OnCopyItems(cut=True)

    if matches:
        if matches.get('mysql'):	
            self.OnPasteItems(L=matches['mysql'])
        else:
            self.OnPasteItems(L=matches['sqlite'])
    else:
        cursor = self.Cursors[LOCAL_HOST]

        for item in self.copyitems:
            z = item.owners+[None,None,None]
            id = self.GetUID() #give it a new id

            #need this or else it won't be seen as a new item when syncing; would be seen as updated
            createdate = mx.DateTime.now() 
            command = "INSERT INTO "+table+" (priority,name,createdate,finisheddate,duedate,note,owner1,owner2,owner3,id) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)"
            cursor.execute(command,(item.priority,item.name,createdate,item.finisheddate,item.duedate,item.notes,z[0],z[1],z[2],id))
            timestamp = self.TimeStamper(host, cursor, table, id)

    self.copyitems = []



.. @+node:ekr.20161117164321.479: *10* Change/update items methods
.. @+node:ekr.20161117164321.480: *11* OnToggleFinished
def OnToggleFinished(self, evt=None):
    L = self.L
    LCtrl = self.ListCtrls[L]
    Properties = self.PropertyDicts[L]
    idx = self.curIdx

    item = self.ItemLists[L][idx]
    LC_Item = LCtrl.GetItem(idx)

    if not item.finisheddate:
        item.finisheddate = mx.DateTime.today()
        LC_Item.SetImage(LCtrl.idx0)
    else:
        item.finisheddate = None
        LC_Item.SetImage(LCtrl.idx1)

    << draw item >>

    self.tb.EnableTool(30, True)

    host = Properties['host']	
    cursor = self.Cursors[host]
    table = Properties['table']

    cursor.execute("UPDATE "+table+" SET finisheddate = %s WHERE id = %s", (item.finisheddate, item.id))
    item.timestamp = self.TimeStamper(host, cursor, table, item.id)

    if Properties['LCdate'] == 'timestamp':
        LCtrl.SetStringItem(idx, self.attr2col_num['date'], item.timestamp.Format("%m/%d %H:%M:%S"))
    elif Properties['LCdate'] == 'finisheddate':
        LCtrl.SetStringItem(idx, self.attr2col_num['date'], item.finisheddate.Format('%m/%d/%y'))



.. @+node:ekr.20161117164321.481: *12* << draw item >>
if item.finisheddate:
    #It appears that SetTextColour resets font weight to Normal but this makes no sense
    #This means that all finished items have Normal weight whether they are priority 3,2 or 1
    #May actually be that GetItem() and then SetItem() sets the weight to Normal no matter what it was originally
    LC_Item.SetTextColour(wxLIGHT_GREY)

elif item.priority==1:
    #see note above about SetTextColour apparently resetting weight
    LC_Item.SetTextColour(wxBLACK)

elif item.priority==2:
    #LC_Item.SetTextColour(wxBLACK) -- this line should be necessary but it does not appear to be
    # ? font is black so ? if have to reset it
    f = self.LC_font
    f.SetWeight(wxBOLD)
    LC_Item.SetFont(f)
    f.SetWeight(wxNORMAL) # resetting font weight

else:
    LC_Item.SetTextColour(wxRED) #appears to be the only way to set color - can't through font
    f = self.LC_font #LCtrl.font
    f.SetWeight(wxBOLD)
    LC_Item.SetFont(f)
    f.SetWeight(wxNORMAL) # resetting font weight

LCtrl.SetItem(LC_Item)
.. @+node:ekr.20161117164321.482: *11* OnPriority
def OnPriority(self, event=None, input=None):
    L = self.L
    idx = self.curIdx
    LCtrl = self.ListCtrls[L]
    Properties = self.PropertyDicts[L]
    item = self.ItemLists[L][idx]

    if input:
        item.priority=input

    else:
        if item.priority < 3:
            item.priority+= 1
        else:
            item.priority=1

    LC_Item = LCtrl.GetItem(idx)

    << draw item >>

    text = str(item.priority)        
    LCtrl.SetStringItem(idx, 0, text)

    host = Properties['host']
    cursor = self.Cursors[host]
    table = Properties['table']

    cursor.execute("UPDATE "+table+" SET priority = %s WHERE id = %s", (item.priority,item.id))
    item.timestamp = self.TimeStamper(host, cursor, table, item.id)

    if Properties['LCdate'] == 'timestamp':
        LCtrl.SetStringItem(idx, self.attr2col_num['date'], item.timestamp.Format('%m/%d %H:%M:%S'))

    wxCallAfter(LCtrl.SetFocus)

.. @+node:ekr.20161117164321.483: *12* << draw item >>
if item.finisheddate:
    #It appears that SetTextColour resets font weight to Normal but this makes no sense
    #This means that all finished items have Normal weight whether they are priority 3,2 or 1
    #May actually be that GetItem() and then SetItem() sets the weight to Normal no matter what it was originally
    LC_Item.SetTextColour(wxLIGHT_GREY)

elif item.priority==1:
    #see note above about SetTextColour apparently resetting weight
    LC_Item.SetTextColour(wxBLACK)

elif item.priority==2:
    #LC_Item.SetTextColour(wxBLACK) -- this line should be necessary but it does not appear to be
    # ? font is black so ? if have to reset it
    f = self.LC_font
    f.SetWeight(wxBOLD)
    LC_Item.SetFont(f)
    f.SetWeight(wxNORMAL) # resetting font weight

else:
    LC_Item.SetTextColour(wxRED) #appears to be the only way to set color - can't through font
    f = self.LC_font #LCtrl.font
    f.SetWeight(wxBOLD)
    LC_Item.SetFont(f)
    f.SetWeight(wxNORMAL) # resetting font weight

LCtrl.SetItem(LC_Item)
.. @+node:ekr.20161117164321.484: *11* Inplace Edit Methods
.. @+node:ekr.20161117164321.485: *12* OnDisplayInPlaceEditor
def OnDisplayInPlaceEditor(self,evt=None):
    L = self.L
    LCtrl = self.ListCtrls[L]
    Properties = self.PropertyDicts[L]
    idx = self.curIdx
    item = self.ItemLists[L][idx]

    host = Properties['host']
    cursor = self.Cursors[host]
    table = Properties['table']

    #if self.Conflict(host, cursor, table, item): return #works -- may be overkill so i've commented it out

    TCid = wxNewId()
    y = LCtrl.GetItemPosition(idx)[1] 
    length = LCtrl.GetColumnWidth(1)

    editor = wxTextCtrl(self, TCid, pos = (167,y+28), size = (length,23), style=wxTE_PROCESS_ENTER)
    editor.SetFont(wxFont(9, wxSWISS, wxNORMAL, wxNORMAL))
    editor.SetBackgroundColour(wxColour(red=255,green=255,blue=175)) #Yellow
    editor.AppendText(item.name)
    editor.Show(True)
    editor.Raise()
    editor.SetSelection(-1,-1)
    editor.SetFocus()	

    EVT_TEXT_ENTER(self, TCid, self.OnCloseInPlaceEditor)		

    self.in_place_editor = editor
    self.modified['inplace'] = 1	





.. @+node:ekr.20161117164321.486: *12* OnCloseInPlaceEditor
def OnCloseInPlaceEditor(self,evt=None):
    L = self.L
    LCtrl = self.ListCtrls[L]
    Properties = self.PropertyDicts[L]
    idx = self.curIdx
    item = self.ItemLists[L][idx]

    host = Properties['host']
    cursor = self.Cursors[host]
    table = Properties['table']
    LCdate = Properties['LCdate']

    #if self.Conflict(host, cursor, table, item)...

    text = self.in_place_editor.GetValue().strip()[:150]
    item.name = text
    LCtrl.SetStringItem(idx, self.attr2col_num['name'], text)
    self.in_place_editor.Destroy()

    cursor.execute("UPDATE "+table+" SET name = %s WHERE id = %s", (text, item.id))
    item.timestamp = self.TimeStamper(host, cursor, table, item.id)

    if Properties['LCdate'] == 'timestamp':
        LCtrl.SetStringItem(idx, self.attr2col_num['date'], item.timestamp.Format('%m/%d %H:%M:%S'))

    self.name.Clear()
    self.name.AppendText(text) #this will cause self.modified['name'] = 1, which is dealt with below

    #using default in case for some reason self.modified does not have the keys
    self.modified.pop('inplace', None)
    self.modified.pop('name', None)

    wxCallAfter(LCtrl.SetFocus) #sets focus on LCtrl and current selection to be highlighted



.. @+node:ekr.20161117164321.487: *11* OnDueDate
def OnDueDate(self, evt=None):
    idx = self.curIdx
    if idx == -1:
        return
    L = self.L
    Properties = self.PropertyDicts[L]
    item = self.ItemLists[L][idx]
    LCtrl = self.ListCtrls[L]

    if item.duedate:
        date = wxDateTime()
        date.SetTimeT(item.duedate) #I am surprised it takes a mx.DateTime object; supposed to need ticks
    else:
        date = 0
    dlg = CalendarDialog(parent=self,
                 title="Select a date",
                 size=(400,400),
                 style=wxCAPTION,
                 date = date)
    if dlg.ShowModal()==wxID_OK:
        date = dlg.cal.GetDate() # this is some date object
        #date = date.GetTicks()
        item.duedate = mx.DateTime.DateFromTicks(date.GetTicks())

        host = Properties['host']
        cursor = self.Cursors[host]
        table = Properties['table']

        cursor.execute("UPDATE "+table+" SET duedate = %s WHERE id = %s", (item.duedate,item.id))
        item.timestamp = self.TimeStamper(host, cursor, table, item.id)
        if Properties['LCdate'] == 'timestamp':
            LCtrl.SetStringItem(idx, self.attr2col_num['date'], item.timestamp.Format("%m/%d %H:%M:%S"))
        elif Properties['LCdate'] == 'duedate':
            LCtrl.SetStringItem(idx, self.attr2col_num['date'], item.duedate.Format('%m/%d/%y'))
    dlg.cal.Destroy()
    dlg.Destroy()

.. @+node:ekr.20161117164321.488: *11* OnEditOwner
def OnEditOwner(self, evt=None): #, new=False) removed Aug. 31 for simplicity
    idx = self.curIdx
    if idx == -1:
        return
    L = self.L
    Properties = self.PropertyDicts[L]
    LCtrl = self.ListCtrls[L]
    item = self.ItemLists[L][idx]
    if not self.ModifierDialog:
        g.pr("self.ModifierDialog is still being constructed")
        return
    #need to clear the current selections or you'll just be making more and more selections
    self.ModifierDialog.SelectCurrent(item.owners)
    self.ModifierDialog.tc.Clear()
    self.ModifierDialog.CenterOnParent()

    val = self.ModifierDialog.ShowModal()

    if val == wxID_OK:
        item.owners, new_names = self.ModifierDialog.GetUserInput()

        << Common Owner Code >>

        for owner in item.owners:
            if self.OwnerLBoxes[L].FindString(owner) == -1:
                self.OwnerLBoxes[L].Append(owner)

        for owner in new_names:
            self.ModifierDialog.lb.Append(owner)

        host = Properties['host']
        cursor = self.Cursors[host]
        table = Properties['table']

        cursor.execute("UPDATE "+table+" SET owner1 = %s, owner2 = %s, owner3 = %s WHERE id = %s", (z[0],z[1],z[2],item.id))
        item.timestamp = self.TimeStamper(host, cursor, table, item.id)
        if Properties['LCdate'] == 'timestamp':
            LCtrl.SetStringItem(idx, self.attr2col_num['date'], item.timestamp.Format("%m/%d %H:%M:%S"))

        if 'owners' in self.modified:
            del self.modified['owners']

    wxCallAfter(LCtrl.SetFocus)

.. @+node:ekr.20161117164321.489: *12* << Common Owner Code >>
owner_str = '; '.join(item.owners)
LCtrl.SetStringItem(idx, self.attr2col_num['owners'], owner_str)
self.owners.Clear()
self.owners.AppendText(owner_str)

z = item.owners+[None,None,None] #note that + creates a new list
.. @+node:ekr.20161117164321.490: *11* OnUpdate
def OnUpdate(self, evt=None):
    if 'inplace' in self.modified:
        self.OnCloseInPlaceEditor()
        if not self.modified:
            return

    L = self.L
    LCtrl = self.ListCtrls[L]
    IList = self.ItemLists[L]
    Properties = self.PropertyDicts[L]
    OLBox = self.OwnerLBoxes[L]
    idx = self.curIdx

    # there is some chance that it is never true that idx == -1 and then this could be eliminated
    if idx != -1:
        item = IList[idx]
    else:
        msg = wxMessageDialog(self, "There is no selected item to update", "", wxICON_ERROR|wxOK)
        msg.ShowModal()
        msg.Destroy()
        self.modified = {}
        return

    host = Properties['host']
    cursor = self.Cursors[host]
    table = Properties['table']

    if 'name' in self.modified:
        item.name = self.name.GetValue().strip()[:150]
        LCtrl.SetStringItem(idx, self.attr2col_num['name'], item.name)
        cursor.execute("UPDATE "+table+" SET name =%s WHERE id = %s",(item.name,item.id))

    if 'note' in self.modified:
        note = self.note.GetValue() #a blank note starts out as None but after this it becomes '' -- ??
        cursor.execute("UPDATE "+table+" SET note =%s WHERE id = %s",(note,item.id))

    if 'owners' in self.modified:
        owner_str = self.owners.GetValue().strip()
        item.owners = []
        if owner_str:
            owner_list = [x.strip() for x in owner_str.split(';')]
            for owner in owner_list:
                owner = ", ".join([x.strip().title() for x in owner.split(',')])
                item.owners.append(owner)

        << Common Owner Code >>

        cursor.execute("UPDATE "+table+" SET owner1 = %s, owner2 = %s, owner3 = %s WHERE id = %s", (z[0],z[1],z[2],item.id))

        for owner in item.owners:
            if self.ModifierDialog.lb.FindString(owner) == -1:
                self.ModifierDialog.lb.Append(owner)
                OLBox.Append(owner)
            elif OLBox.FindString(owner) == -1:
                OLBox.Append(owner)		

    item.timestamp = self.TimeStamper(host, cursor, table, item.id)
    if Properties['LCdate'] == 'timestamp':
        LCtrl.SetStringItem(idx, 3, item.timestamp.Format("%m/%d %H:%M:%S"))

    self.modified = {}


.. @+node:ekr.20161117164321.491: *12* << Common Owner Code >>
owner_str = '; '.join(item.owners)
LCtrl.SetStringItem(idx, self.attr2col_num['owners'], owner_str)
self.owners.Clear()
self.owners.AppendText(owner_str)

z = item.owners+[None,None,None] #note that + creates a new list
.. @+node:ekr.20161117164321.492: *11* OnNewItem
def OnNewItem(self, evt=None):
    L=self.L
    LCtrl = self.ListCtrls[L]
    Properties = self.PropertyDicts[L]

    if self.curIdx != -1:
        LCtrl.SetItemState(self.curIdx, 0, wxLIST_STATE_SELECTED)

    << Clear data fields >>

    class Item: pass
    item = Item()
    item.name = '<New Item>'
    item.priority = 1
    item.owners = []
    item.createdate = mx.DateTime.now() #need this to be a timestamp and not just date for syncing
    item.duedate = item.finisheddate = None

    self.ItemLists[L].insert(0,item)

    host = Properties['host']
    cursor = self.Cursors[host]
    table = Properties['table']
    item.id = self.GetUID()

    cursor.execute("INSERT INTO "+table+" (priority,name,createdate,finisheddate,duedate,id) VALUES (%s,%s,%s,%s,%s,%s)",
                (item.priority,item.name,item.createdate,None,None,item.id))

    item.timestamp = self.TimeStamper(host, cursor, table, item.id)

    #tracking new item for syncing will happen in Edit Name

    LCtrl.InsertImageStringItem(0,"1", LCtrl.idx1)
    LCtrl.SetStringItem(0,1,item.name)

    if Properties['LCdate'] == 'timestamp':
        LCtrl.SetStringItem(0, self.attr2col_num['date'], item.timestamp.Format("%m/%d %H:%M:%S"))
    elif Properties['LCdate'] == 'createdate':
        LCtrl.SetStringItem(0, self.attr2col_num['date'], item.createdate.Format('%m/%d/%y'))

    self.curIdx = 0

    #if Display is being filtered we assume that is the owner of the new node
    owner = Properties['owner']	
    if owner and owner!='*ALL':
        self.ListCtrls[L].SetStringItem(0, self.attr2col_num['owners'], owner)
        item.owners = [owner]

        self.owners.Clear()
        self.owners.AppendText(owner)

        cursor.execute("UPDATE "+table+" SET owner1 = %s WHERE id = %s", (owner,item.id))
        item.timestamp = self.TimeStamper(host, cursor, table, item.id)  #not really necessary since just got a timestamp

    # decided that it was actually better not to ask for the owner on a new node	
    #else:
        #self.OnEditOwner()

    LCtrl.SetFocus() #needed for the in place editor to look right
    LCtrl.SetItemState(0, wxLIST_STATE_SELECTED, wxLIST_STATE_SELECTED)

    self.OnDisplayInPlaceEditor() #(new=True)
.. @+node:ekr.20161117164321.493: *12* << Clear data fields >>
self.name.Clear()
self.owners.Clear()
self.note.Clear()
.. @+node:ekr.20161117164321.494: *11* Conflict (not in use)
@ Need to decide if we are going to have timestamp checking to be sure something hasn't changed
Note that there would not need to be timestamp checking on a new node
Also  there is no need to timestamp check on a local DB
The following code seems to work fine, however, I have just commented out the calls to it in NameEditor methods
@c
def Conflict(self, host, cursor, table, item):
    if host is 'sqlite':
        return False
    cursor.execute("Select timestamp from "+table+" WHERE id = %s", (item.id,))
    db_timestamp = cursor.fetchone()[0]
    if db_timestamp != item.timestamp:
        g.pr("There is a conflict and you should refresh display")
        return True
    else:
        return False
.. @+node:ekr.20161117164321.495: *11* OnEditNote
def OnEditNote(self, evt=None):
    if self.modified:
        self.OnUpdate()

    idx = self.curIdx

    if idx == -1:
        return

    L = self.L

    #if self.editor:
        #machine = None
        #win32pdh.EnumObjects(None, machine, 0, 1) # resets Enum otherwise it seems to hold onto old data
        #object = "Process"
        #items, instances = win32pdh.EnumObjectItems(None,None,"Process", -1)
        #if 'TextPad' in instances:
            #g.pr("TextPad is running")
        #else:
            #self.editor = {}

    item = self.ItemLists[L][idx]
    file_name = re.sub('[\\/:*"<>|\?]','-',item.name) #make sure all chars are legal file name characters

    path = os.path.join(os.environ['TMP'],file_name[:50])+'.%s'%NOTE_EXT

    f = file(path,'w')
    f.write(self.GetNote())
    f.close()

    os.startfile(path)

    id = item.id
    for d in self.editor:
        if d['id'] == id:
            return

    ed = {}
    ed['time'] = os.path.getmtime(path)
    ed['host'] = self.PropertyDicts[L]['host']
    ed['table'] = self.PropertyDicts[L]['table']
    ed['path'] = path
    ed['id'] = item.id

    self.editor.append(ed)

    time.sleep(.1)
.. @+node:ekr.20161117164321.496: *10* File menu methods
.. @+node:ekr.20161117164321.497: *11* OnNewList
def OnNewList(self, event=None):
    if self.modified:
        self.OnUpdate()

    if OFFLINE_ONLY is True or REMOTE_HOST is None:
        hosts = [LOCAL_HOST]
    else:
        hosts = [LOCAL_HOST, REMOTE_HOST]

    dlg = wxSingleChoiceDialog(self, 'Databases', 'Choose a database:', hosts, wxCHOICEDLG_STYLE)
    val = dlg.ShowModal()
    dlg.Destroy()
    if val == wxID_OK:
        host = dlg.GetStringSelection()
    else:
        return

    cursor = self.GetCursor(host)
    if cursor is None:
        return

    dlg = wxTextEntryDialog(self, 'What is the name of the new table?', 'Create Table')
    val = dlg.ShowModal()
    dlg.Destroy()
    if val == wxID_OK:
        table = dlg.GetValue()
    else:
        return

    if not table:
        return

    location, rdbms = host.split(':')

    if rdbms == 'sqlite':
        cursor.execute("SELECT name FROM sqlite_master WHERE type='table' ORDER BY name")
    else:
        cursor.execute("SHOW tables")

    if (table,) in cursor.fetchall():
        msg = wxMessageDialog(self,
                              "Table '%s' already exists"%table,
                              "Duplicate Table",
                              wxICON_ERROR|wxOK)
        msg.ShowModal()
        msg.Destroy()
        return

    dlg = wxMessageDialog(self,
          "Are you sure you want to create Table '%s'?"%table,
          "Create Table?",
          wxICON_QUESTION|wxYES_NO)

    if dlg.ShowModal() == wxID_YES:
        self.CreateTable(host,table)
        self.CreateNewNotebookPage(host,table)

        #self.AddListControl(tab_title) #add listcontrol displays the list

        #self.OnNewItem()

    dlg.Destroy()


.. @+node:ekr.20161117164321.498: *11* OnFileList
def OnFileList(self, evt=None, path=None):
    if self.modified:
        self.OnUpdate()

    #if there is no event, we got here through the start up loading of lists
    if evt:
        fileNum = evt.GetId() - wxID_FILE1			
        path = self.filehistory.GetHistoryFile(fileNum)
        location, rdbms, table = path.split(':')
        host = '%s:%s'%(location, rdbms)
        # only need to check if table is open if this is not at startup
        if table in [p['table'] for p in self.PropertyDicts if p['host'] == host]:
            dlg = wxMessageDialog(self,"%s (%s) is already open!"%(table,host),"List Open",wxICON_ERROR|wxOK)
            dlg.ShowModal()
            dlg.Destroy()
            return

    else:
        location, rdbms, table = path.split(':')
        host = '%s:%s'%(location, rdbms)

    cursor = self.GetCursor(host)
    if cursor is None:
        return

    if rdbms == 'sqlite':
        sql = "SELECT name FROM sqlite_master WHERE name = '%s'"%table
    else:
        sql = "SHOW TABLES LIKE '%s'"%table

    cursor.execute(sql)
    if not cursor.fetchall():
        dlg = wxMessageDialog(self,
                    "Table '%s' at host '%s' does not appear to exist!"%(table,host),
                    "Table does not exist",
                    wxICON_ERROR|wxOK)
        dlg.ShowModal()
        dlg.Destroy()
        return

    self.CreateNewNotebookPage(host,table)

.. @+node:ekr.20161117164321.499: *11* OnOpenList
def OnOpenList(self, evt=None):
    if self.modified:
        self.OnUpdate()

    tree = {}

    if OFFLINE_ONLY is True or REMOTE_HOST is None:
        hosts = [LOCAL_HOST]
    else:
        hosts = [LOCAL_HOST, REMOTE_HOST]

    for host in hosts:
        cursor = self.GetCursor(host)
        if cursor:
            if host.split(':')[1] == 'sqlite':
                sql = "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"
            else:
                sql = "SHOW TABLES" #sorted

            cursor.execute(sql)
            results = cursor.fetchall()

            #excluding already open tables + 'system' tables
            excluded_tables = [p['table'] for p in self.PropertyDicts if p['host'] == host]
            excluded_tables.extend(['user_sync','sync','owners'])

            tables = [t for (t,) in results if t not in excluded_tables]

            tree[host] = tables

    dlg = TreeDialog(self, "Open List", tree=tree)
    val = dlg.ShowModal()
    dlg.Destroy()
    if val == wxID_OK:
        sel = dlg.TreeCtrl.GetSelection()
        table = dlg.TreeCtrl.GetItemText(sel)
        sel = dlg.TreeCtrl.GetItemParent(sel)
        host = dlg.TreeCtrl.GetItemText(sel)

        if host in hosts: #takes care of highlighting root or hosts
            self.CreateNewNotebookPage(host,table)
.. @+node:ekr.20161117164321.500: *11* OnDeleteList
def OnDeleteList(self, evt=None):
    #ini controls whether the menu item is enabled
    Properties = self.PropertyDicts[self.L]
    host = Properties['host']
    table = Properties['table']

    #if table is in SYNC_TABLES, should we make a point of that?
    dlg = wxMessageDialog(self,
                        "Are you sure that you want to delete table %s (%s)?\n(Please note that you cannot recover it once it is deleted!)"%(table,host),
                        "Delete Table...",
                        wxICON_EXCLAMATION|wxYES_NO|wxNO_DEFAULT)

    val = dlg.ShowModal()
    dlg.Destroy()
    if val == wxID_NO:
        return

    rdbms = host.split(':')[1]

    if rdbms == 'mysql':
        dlg = wxMessageDialog(self,
                        "Are you sure really really sure you want to delete table %s (%s)?\n(You really really cannot recover it once it is deleted)"%(table,host),
                        "Delete Table...",
                        wxICON_EXCLAMATION|wxYES_NO|wxNO_DEFAULT)

        val = dlg.ShowModal()
        dlg.Destroy()
        if val == wxID_NO:
            return

    cursor = self.Cursors[host]
    cursor.execute("DROP TABLE %s"%table)

    self.OnCloseList()

.. @+node:ekr.20161117164321.501: *11* OnCloseList
def OnCloseList(self, evt=None):
    if self.modified:
        self.OnUpdate()

    L = self.L

    del self.ItemLists[L]
    del self.PropertyDicts[L]
    del self.ListCtrls[L]
    del self.OwnerLBoxes[L]

    self.nb.DeletePage(L)        

    ln = len(self.PropertyDicts)
    if ln:
        self.nb.SetSelection(0)
        self.L = 0
    else:
        self.L = -1




.. @+node:ekr.20161117164321.502: *11* OnCloseAll
def OnCloseAll(self, evt=None):
    if self.modified:
        self.OnUpdate()

    while self.L != -1:
        self.OnCloseList()

    self.name.Clear()
    self.owners.Clear()
    self.note.Clear()
    #note that Clearing does set self.modified (eg {'name':1})
    self.modified = {}

.. @+node:ekr.20161117164321.503: *11* OnSaveAsText
def OnSaveAsText(self, evt=None):
    if self.modified:
        self.OnUpdate()

    Properties = self.PropertyDicts[self.L]
    wildcard = "txt files (*.txt)|*.txt|All files (*.*)|*.*"
    #dlg = wxFileDialog(self, "Save file", "", Properties['table'], wildcard, wxSAVE|wxOVERWRITE_PROMPT|wxCHANGE_DIR)

    body = ""
    for i,item in enumerate(self.ItemLists[self.L]):
        body = body+"%d. %s (%d)\n"%(i+1, item.name, item.priority)

    table = Properties['table']
    location, rdbms = Properties['host'].split(':')
    filename = re.sub('[\\/:*"<>|\?]','-','%s-%s-%s'%(location,rdbms,table)) 
    filename = filename[:50]+'.txt'

    path = os.path.join(DIRECTORY,filename)

    f = file(path,'w')
    f.write(body)
    f.close()

    os.startfile(path)

    self.SetStatusText("Saved file %s"%path)

.. @+node:ekr.20161117164321.504: *11* OnArchive
def OnArchive(self, evt=None):
    if self.modified:
        self.OnUpdate()

    Properties = self.PropertyDicts[self.L]
    host = Properties['host']
    cursor = self.Cursors[host]
    table = Properties['table']
    rdbms = host.split(':')[1]

    table_archive = table+'_archive'

    #need to test for existence of table_archive
    if rdbms == 'sqlite':
        cursor.execute("SELECT name FROM sqlite_master WHERE type='table' ORDER BY name")
    else:
        cursor.execute("SHOW tables")

    results = cursor.fetchall()

    if (table_archive,) not in results:
        dlg = wxMessageDialog(self,
                    "Do you want to create an archive for table %s (%s)"%(table,rdbms),
                    "Create an archive...",
                    wxICON_QUESTION|wxYES_NO)
        val = dlg.ShowModal()
        dlg.Destroy()
        if val==wxID_YES:
            self.CreateTable(host,table_archive)
        else:
            return

    label1 = "In table %s (%s) \narchive all finished items older than:"%(table,rdbms)
    label2 = "Archive all finished items"
    dlg = FinishedDialog(self, "Archive completed items", days=7, spin_label=label1, check_label=label2)

    val = dlg.ShowModal()
    dlg.Destroy() #dialogs and frames not destroyed right away to allow processing events, methods
    if val==wxID_CANCEL:
        return

    if dlg.check.GetValue():
        cursor.execute("SELECT id,priority,name,createdate,finisheddate,duedate,owner1,owner2,owner3,note FROM "+table+" WHERE finisheddate IS NOT NULL")
    else:
        days = dlg.text.GetValue()
        date = mx.DateTime.today() - int(days)
        cursor.execute("SELECT id,priority,name,createdate,finisheddate,duedate,owner1,owner2,owner3,note FROM "+table+" WHERE finisheddate < %s",(date,))

    results = cursor.fetchall()
    dlg = wxMessageDialog(self,
                        "Archiving will remove %d records from %s.\nDo you want to proceed?"%(len(results),table),
                        "Proceed to archive...",
                        wxICON_QUESTION|wxYES_NO)

    val = dlg.ShowModal()
    dlg.Destroy()
    if val == wxID_NO:
        return

    if table in SYNC_TABLES:
        if rdbms == 'sqlite':
            def track_deletes():
                timestamp = mx.DateTime.now()
                cursor.execute("INSERT INTO sync (id,action,table_name,name,timestamp) VALUES (%s,%s,%s,%s,%s)",(id,'d',table,name,timestamp))
        else:
            def track_deletes():
                cursor.execute("INSERT INTO sync (id,action,table_name,user,name) VALUES (%s,%s,%s,%s,%s)",(id,'d',table,USER,name))
    else:
        def track_deletes():
            pass	

    for row in results:
        # the next line is necessary because pysqlite returns a tuple-like object that is not a tuple
        r = tuple(row)
        id = r[0]
        name = r[2]
        cursor.execute("INSERT INTO "+table_archive+"  (id,priority,name,createdate,finisheddate,duedate,owner1,owner2,owner3,note) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)",r)
        timestamp = self.TimeStamper(host, cursor, table_archive, id)
        cursor.execute("DELETE from "+table+" WHERE id = %s", (id,))
        track_deletes()

    self.OnRefresh()
    dlg = wxMessageDialog(self,
                        "Table %s had items older than %s days successfully archived"%(table,days),
                        "Archiving successful...",
                        wxICON_INFORMATION|wxOK)
    dlg.ShowModal()
.. @+node:ekr.20161117164321.505: *11* OnWorkOffline
def OnWorkOffline(self, evt=None):
    global OFFLINE_ONLY
    OFFLINE_ONLY = not OFFLINE_ONLY
    if OFFLINE_ONLY:
        del self.Cursors[REMOTE_HOST]
    else:
        server = REMOTE_HOST.split(':')[0]
        try:
            socket.gethostbyname(server)
        except:
            dlg = wxMessageDialog(None, "Cannot connect to remote server! Will set to work offline.", "ListManager", style=wxOK|wxICON_EXCLAMATION|wxSTAY_ON_TOP)
            dlg.ShowModal()
            dlg.Destroy()
            OFFLINE_ONLY = True

    self.filemenu.Check(idOFFLINE,OFFLINE_ONLY)

.. @+node:ekr.20161117164321.506: *10* Display methods
.. @+node:ekr.20161117164321.507: *11* OnItemSelected
def OnItemSelected(self, evt=None):
    if self.modified:
        self.OnUpdate()

    if evt:
        idx = evt.GetIndex()
    elif self.curIdx != -1:
        idx = self.curIdx
    else: # really to catch self.curIdx = -1 (see OnDelete and OnRefresh)
        self.name.Clear() # could be moved out of if
        self.owners.Clear() # could be moved out of if
        self.note.Clear()
        #note that Clearing does set self.modified (eg {'name':1})
        self.modified = {}
        return

    L = self.L
    item = self.ItemLists[L][idx]

    self.name.Clear()
    self.name.AppendText(item.name) #SetValue(item.name) - if you use setvalue you don't get the font

    self.owners.Clear()
    self.owners.AppendText('; '.join(item.owners))

    note = self.GetNote(L,item)
    if note.find("<leo_file>") != -1:
        self.note.SetValue("Leo Outline")
        self.note.SetEditable(False)
    else:
        self.note.SetValue(note)
        self.note.SetEditable(True)

    self.ListCtrls[L].EnsureVisible(idx)
    self.curIdx = idx

    #writing to text widgets caused wxEVT_COMMAND_TEXT_UPDATED which is caught by EVT_TEXT, which updates self.modified
    self.modified={}

.. @+node:ekr.20161117164321.508: *11* OnItemActivated
def OnItemActivated(self,evt):
    g.pr("On Activated")

.. @+node:ekr.20161117164321.509: *11* OnShowAll
def OnShowAll(self, evt=None):
    L = self.L
    OLBox = self.OwnerLBoxes[L]

    Properties = self.PropertyDicts[L]
    Properties['showfinished'] = -1
    Properties['owner'] = '*ALL'

    OLBox.SetStringSelection('*ALL')

    self.OnRefresh()
.. @+node:ekr.20161117164321.510: *11* OnRefresh
def OnRefresh(self, evt=None):
    #OnItemSelected should be able to handle no items so this could be very short
    L = self.L

    results = self.ReadFromDB()
    self.ItemLists[L] = self.CreateAndDisplayList(results)

    if self.ItemLists[L]:
        self.ListCtrls[L].SetItemState(0, wxLIST_STATE_SELECTED, wxLIST_STATE_SELECTED)
        self.curIdx = 0
    else:
        self.curIdx = -1		

    self.OnItemSelected()
.. @+node:ekr.20161117164321.511: *11* OnFilterOwners
def OnFilterOwners(self, evt=None):
    if self.modified:
        self.OnUpdate()
    sel = self.OwnerLBoxes[self.L].GetStringSelection()

    if sel:
        self.PropertyDicts[self.L]['owner'] = sel
        self.OnRefresh()
.. @+node:ekr.20161117164321.512: *11* OnColumnClick (to sort columns)
def OnColumnClick(self, evt):
    col_num = evt.GetColumn()
    L = self.L
    LCtrl = self.ListCtrls[L]
    Sort = self.PropertyDicts[L]['sort']
    attr2col = self.attr2col_num

    prev_sort_attr = Sort.get('attribute') #if this is the first sort Properties['sort'] is {}

    #following is a little bit ugly but gets the key from the value, which is col_num
    Sort['attribute'] = attr2col.keys()[attr2col.values().index(col_num)]

    if prev_sort_attr == Sort['attribute']:
        Sort['direction'] = not Sort['direction']
    else:
        Sort['direction'] = 0

    self.OnRefresh()

    LCtrl.ClearColumnImage(attr2col['priority'])
    LCtrl.ClearColumnImage(attr2col['date'])
    img_num = LCtrl.arrows[Sort['direction']]
    LCtrl.SetColumnImage(col_num, img_num)

.. @+node:ekr.20161117164321.513: *11* OnShowFinished
def OnShowFinished(self,evt):
    Properties = self.PropertyDicts[self.L]
    label1 = "Enter the number of days to retain\ncompleted tasks in the display:"
    label2 = "Show all finished items"
    dlg = FinishedDialog(self, "Display of completed items", days=Properties['showfinished'], spin_label=label1, check_label=label2)
    if dlg.ShowModal()==wxID_OK:
        if dlg.check.GetValue():
            Properties['showfinished'] = -1
        else:
            days = dlg.text.GetValue()
            Properties['showfinished'] = int(days)			
        self.OnRefresh()
    dlg.Destroy()

.. @+node:ekr.20161117164321.514: *11* OnColumnRightClick (popup to change date displayed)
def OnColumnRightClick(self, evt=None):
    col = evt.GetColumn()
    if col != self.attr2col_num['date']:
        return

    L = self.L
    LCtrl = self.ListCtrls[L]
    Properties = self.PropertyDicts[L]

    #x,y = evt.GetPosition()
    datemenu = wxMenu()

    for i,date in enumerate(['Create Date','Last Modified','Due Date','Completion Date']):
        datemenu.Append(200+i, date)
        EVT_MENU(self, 200+i, lambda e, i=i: self.ChangeDateDisplayed(e,i))

    x = LCtrl.GetColumnWidth(1)+ LCtrl.GetColumnWidth(2) + LCtrl.GetColumnWidth(3)
    self.PopupMenu(datemenu,(x,40))
    datemenu.Destroy()


.. @+node:ekr.20161117164321.515: *11* OnDisplayDateCategory
def OnDisplayDateCategory(self, evt=None):
    dlg = wxSingleChoiceDialog(self, 'Date Display', 'Choose a date to display:',
                    ['Create Date','Last Modified','Due Date','Completion Date']
                    , wxOK|wxCANCEL)
    val = dlg.ShowModal()
    dlg.Destroy()

    if val == wxID_OK:
        idx = dlg.GetSelection()
        self.ChangeDateDisplayed(i=idx)

.. @+node:ekr.20161117164321.516: *11* ChangeDateDisplayed
def ChangeDateDisplayed(self, evt=None, i=0):
    L = self.L
    LCtrl = self.ListCtrls[L]
    self.PropertyDicts[L]['LCdate'] = displaydate = ('createdate','timestamp','duedate','finisheddate')[i]	
    col_num = self.attr2col_num['date']
    col_info = LCtrl.GetColumn(col_num)
    col_info.SetText(self.date_titles[displaydate])
    LCtrl.SetColumn(col_num,col_info)
    self.DisplayList(self.ItemLists[L])
    #self.OnRefresh() #have gone back and forth but think that it should be self.DisplayList
.. @+node:ekr.20161117164321.517: *11* DisplayList
def DisplayList(self, List, L=None):
    #OnPasteItems needs to be able to have an L that is not self.L
    if L is None:
        L = self.L
    LCtrl = self.ListCtrls[L]
    LCdate = self.PropertyDicts[L]['LCdate']
    if LCdate == 'timestamp':
        format = '%m/%d %H:%M:%S'
    else:
        format = '%m/%d/%y'
    LCtrl.DeleteAllItems()

    for x,item in enumerate(List):
        << draw item >>


.. @+node:ekr.20161117164321.518: *12* << draw item >>
LCtrl.InsertImageStringItem(x, str(item.priority), LCtrl.idx1)
LCtrl.SetStringItem(x,1,item.name)
LCtrl.SetStringItem(x,2,'; '.join(item.owners))
date = item.__dict__[LCdate]
LCtrl.SetStringItem(x,3,date and date.Format(format) or "")

if item.finisheddate:
    LC_Item = LCtrl.GetItem(x)
    LC_Item.SetImage(LCtrl.idx0) #might just want generic number or greyed one two three
    LC_Item.SetTextColour(wxLIGHT_GREY)
    LCtrl.SetItem(LC_Item)

elif item.priority==2:
    LC_Item = LCtrl.GetItem(x)
    f = self.LC_font
    f.SetWeight(wxBOLD)
    LC_Item.SetFont(f)
    f.SetWeight(wxNORMAL) #resetting weight
    LCtrl.SetItem(LC_Item)

elif item.priority==3:
    LC_Item = LCtrl.GetItem(x)
    f = self.LC_font
    f.SetWeight(wxBOLD)
    LC_Item.SetFont(f)
    f.SetWeight(wxNORMAL) #return to normal
    LC_Item.SetTextColour(wxRED)
    LCtrl.SetItem(LC_Item)
.. @+node:ekr.20161117164321.519: *10* Printing methods
.. @+node:ekr.20161117164321.520: *11* OnPageSetup
def OnPageSetup(self, evt):
    #need to pass printdata to tableprint

    psdata = wxPageSetupDialogData()

    # if want to vary margins will need to save them as ivars and then set
    #psdata.SetMarginTopLeft((self.Left,self.Top))
    psdata.EnableMargins(False)
    psdata.SetPrintData(self.printdata) #gets Paper Orientation and PaperId info from printdata

    dlg = wxPageSetupDialog(self, psdata)
    if dlg.ShowModal() == wxID_OK:
        self.printdata = dlg.GetPageSetupData().GetPrintData()
        dlg.Destroy()
.. @+node:ekr.20161117164321.521: *11* OnPrint
def OnPrint(self, evt=None, prev=False, showprtdlg=True): 		#???self.psdata = psdata
    IList = self.ItemLists[self.L]
    Properties = self.PropertyDicts[self.L]

    prt = PrintTable(self.printdata) #self.printdata is the wxPrintData object with Orientation Info

    font_name = prt.default_font_name
    prt.text_font = {'Name':font_name, 'Size':11, 'Colour':[0, 0, 0], 'Attr':[0, 0, 0]}
    prt.label_font = {'Name':font_name, 'Size':12, 'Colour':[0, 0, 0], 'Attr':[1, 0, 0]}
    prt.header_font = {'Name':font_name, 'Size':14, 'Colour':[0, 0, 0], 'Attr':[1, 0, 0]}

    prt.row_def_line_colour = wxLIGHT_GREY
    prt.column_def_line_colour = wxLIGHT_GREY

    prt.left_margin = 0.5

    data = []
    for row,item in enumerate(IList):	
        data.append([str(item.priority),
                    item.name,
                    item.duedate and item.duedate.Format('%m/%d/%y') or '',
                    '; '.join([x.split(',')[0] for x in item.owners])]) #just last names

        if item.finisheddate:
            prt.SetCellText(row, 0, wxLIGHT_GREY)
            prt.SetCellText(row, 1, wxLIGHT_GREY)
            prt.SetCellText(row, 2, wxLIGHT_GREY)
            prt.SetCellText(row, 3, wxLIGHT_GREY)

    prt.data = data
    prt.label = ['P','Item','Due','Owner']

    if self.printdata.GetOrientation() == wxPORTRAIT:
        prt.set_column = [.2, 5, .65, 1]
    else:
        prt.set_column = [.2, 7, .65, 1.5]

    title = "Table: %s   Owner: %s    "%(Properties['table'],Properties['owner'])
    prt.SetHeader(title, type='Date & Time', align=wxALIGN_LEFT, indent = 1.5)
    prt.SetFooter("Page No ", type ="Num")

    if prev:
        prt.Preview()
    else:
        prt.Print(prompt=showprtdlg)
.. @+node:ekr.20161117164321.522: *10* Exiting methods
.. @+node:ekr.20161117164321.523: *11* OnWindowExit
def OnWindowExit(self, evt):
    #this is called if you close the ListManager Window with the X
    if evt.CanVeto():
        self.OnExit()
    else:
        evt.Skip()
.. @+node:ekr.20161117164321.524: *11* OnExit
def OnExit(self, event=None):   
    << save configuration file >>
    sys.stderr.dlg.Destroy() #destroys the error dialog; need to do this to shut down correctly
    if self.ModifierDialog: #only reason to check is if closed before ModifierDialog is constructed
        self.ModifierDialog.Destroy()
    self.Close(1)
.. @+node:ekr.20161117164321.525: *12* <<save configuration file>>
cp.remove_section('Files')
cp.add_section("Files")

x,y = self.GetSizeTuple()

cp.set('Configuration','x', str(x))
cp.set('Configuration','y', str(y))

numfiles = self.filehistory.GetNoHistoryFiles()

for n in range(numfiles):
    cp.set("Files", "path%d"%n, self.filehistory.GetHistoryFile(n))

try:
    #you have to give ConfigParser a writable object
    cfile = file(config_file, 'w')
    cp.write(cfile)
    cfile.close()
except IOError:
    g.pr("The configuration file can't be written!")
    time.sleep(10) #so you can see that there was a problem
.. @+node:ekr.20161117164321.526: *10* Find methods
.. @+node:ekr.20161117164321.527: *11* OnFind
def OnFind(self, evt=None):
    self.FindDialog.Show(True)
    self.FindDialog.FindText.SetSelection(-1,-1)
    self.FindDialog.FindText.SetFocus()


.. @+node:ekr.20161117164321.528: *11* FindString
def FindString(self, evt=None):
    L = self.L
    Properties = self.PropertyDicts[L]
    cursor = self.Cursors[Properties['host']]
    table = Properties['table']

    pat = self.FindDialog.FindText.GetValue()
    likepat = r"'%"+pat+r"%'"
    finished = self.FindDialog.SearchFinished.GetValue()
    notes = self.FindDialog.SearchNotes.GetValue()

    if finished:
        WHERE = "WHERE "
    else:
        WHERE = "WHERE finisheddate IS NULL AND "

    if notes:
        SELECT = "SELECT priority,name,createdate,finisheddate,duedate,owner1,owner2,owner3,id,timestamp,note FROM %s "%table
        WHERE = WHERE + "(name LIKE %s OR note LIKE %s) ORDER BY timestamp DESC"%(likepat,likepat)
    else:
        SELECT = "SELECT priority,name,createdate,finisheddate,duedate,owner1,owner2,owner3,id,timestamp FROM %s "%table
        WHERE = WHERE + "name LIKE %s ORDER BY timestamp DESC"%likepat

    sql = SELECT + WHERE			
    try:
        cursor.execute(sql)
    except:
        g.pr("Cannot read %s: %s"%(Properties['host'],table))
        return
    else:
        results = cursor.fetchall()

    case = self.FindDialog.MatchCase.GetValue()
    whole = self.FindDialog.MatchWhole.GetValue()

    if whole:
        pat = '\\b%s\\b'%pat

    if case:
        z = re.compile(pat)
    else:
        z =re.compile(pat, re.I)

    if notes:
        results = [x for x in results if re.search(z,x[1]) or re.search(z,x[10])]
    else:
        results = [x for x in results if re.search(z,x[1])]

    Properties['LCdate'] = 'timestamp'
    self.ItemLists[L]= IList = self.CreateAndDisplayList(results)

    LCtrl = self.ListCtrls[L]
    col_num = self.attr2col_num['date']
    col_info = LCtrl.GetColumn(col_num)
    col_info.SetText(self.date_titles['timestamp'])
    LCtrl.SetColumn(col_num,col_info)

    if IList:
        self.curIdx = 0
        LCtrl.SetItemState(0, wxLIST_STATE_SELECTED, wxLIST_STATE_SELECTED)
    else:		
        self.curIdx = -1

    self.OnItemSelected()

    Properties['sort'] = {'direction':0,'attribute':'date'}
    Properties['owner'] = '*ALL'

    owner_idx = self.OwnerLBoxes[L].GetSelection()
    if owner_idx != -1:
        self.OwnerLBoxes[L].SetSelection(owner_idx, 0) #get exception if index = -1

    self.SetStatusText("Found %d items"%len(IList))
.. @+node:ekr.20161117164321.529: *11* FindNode
def FindNode(self, item, showfinished=True):
    L = self.L
    LCtrl = self.ListCtrls[L]
    Properties = self.PropertyDicts[L]

    Properties['owner'] = '*ALL'
    Properties['showfinished'] = showfinished

    self.ItemLists[L] = IList = self.CreateAndDisplayList(self.ReadFromDB())

    id = item.id
    idx = -1
    for item in IList:
        idx+=1
        if id == item.id:
            break
    else:
        idx = -1

    if idx != -1:	
        LCtrl.SetItemState(idx, wxLIST_STATE_SELECTED, wxLIST_STATE_SELECTED)
        LCtrl.EnsureVisible(idx)
    self.curIdx = idx

.. @+node:ekr.20161117164321.530: *10* Database-related methods
.. @+node:ekr.20161117164321.531: *11* GetCursor
def GetCursor(self, host):
    cursor = self.Cursors.get(host)
    if cursor:
        return cursor

    location, rdbms = host.split(':')

    if rdbms == 'sqlite':
        db = os.path.join(DIRECTORY,location,DB)
        try:
            Con = sqlite.connect(db=db, autocommit=1)
            cursor = Con.cursor()
            self.sqlite_connections.append(Con)  #getting a weak reference error from PySQLite and this makes it go away
        except:
            dlg = wxMessageDialog(self,
                    "Could not connect to SQLite database at %s"%location,
                    "Connection problem!",
                    wxICON_ERROR|wxOK)
            dlg.ShowModal()
            dlg.Destroy()
            cursor = None

    elif not OFFLINE_ONLY:
        try:
            Con = MySQLdb.connect(host=location, user=USER, passwd=PW, db=DB)
            cursor = Con.cursor()
        except:
            dlg = wxMessageDialog(self,
                    "host = %s | user = %s | password = %s**** | db = %s - could not connect!"%(host,USER,PW[:3],DB),
                    "Connection problem",
                    wxICON_ERROR|wxOK)
            dlg.ShowModal()
            dlg.Destroy()
            cursor = None

    if cursor:
        self.Cursors[host] = cursor

    return cursor


.. @+node:ekr.20161117164321.532: *11* GetNote
def GetNote(self, L=None, item=None):
    if L is None:
        L = self.L

    if item is None:
        if self.curIdx != -1:
            item = self.ItemLists[L][self.curIdx]
        else:
            return ''

    Properties = self.PropertyDicts[L]

    cursor = self.Cursors[Properties['host']]
    table = Properties['table']
    cursor.execute("SELECT note from "+table+" WHERE id = %s", (item.id,))

    ###### Debug -- this does happen where note brings back None 053003
    z = cursor.fetchone()
    if z is None:
        g.pr("In GetNote -> SELECT should not bring back None")
        g.pr("           -> item.id=",item.id)
        z = (None,)
    note = z[0]
    if note is None:
        note = ''
    return note

.. @+node:ekr.20161117164321.533: *11* CreateTable
def CreateTable(self, host, table):
    cursor = self.Cursors[host]
    rdbms = host.split(':')[1]
    if rdbms == 'sqlite':
        sql = """CREATE TABLE '%s' ('id' varchar(36) PRIMARY KEY,
'priority' int(1),
'name' varchar(150),
'createdate' datetime,
'finisheddate' date,
'duedate' date,
'owner1' varchar(25),
'owner2' varchar(25),
'owner3' varchar(25),
'note' text,
'timestamp' timestamp(14))"""%table
    else:
        sql = """CREATE TABLE `%s` (`id` varchar(36) NOT NULL default '',
`priority` int(1) NOT NULL default '1',
`name` varchar(150) NOT NULL default '',
`createdate` datetime NOT NULL default '0000-00-00 00:00:00',
`finisheddate` date default '0000-00-00',
`duedate` date default '0000-00-00',
`owner1` varchar(25) default '',
`owner2` varchar(25) default '',
`owner3` varchar(25) default '',
`note` text,
`timestamp` timestamp(14) NOT NULL,PRIMARY KEY  (`id`)) TYPE=MyISAM"""%table

    cursor.execute(sql)
.. @+node:ekr.20161117164321.534: *11* ReadFromDB (returns db results)
def ReadFromDB(self):
    L = self.L
    Properties = self.PropertyDicts[L]

    host = Properties['host']
    cursor = self.GetCursor(host)
    if cursor is None:
        return None

    table = Properties['table']

    owner = Properties['owner']
    if owner == '*ALL':
        WHERE = ""
    else:
        WHERE = 'WHERE (owner1 = "%s" OR owner2 = "%s" OR owner3 = "%s")'%(owner,owner,owner)

    #-1 show them all; 0 show none; integer show for that many days
    days = Properties['showfinished']	
    if days != -1:
        if days:
            date = mx.DateTime.now() - days
            t = "(finisheddate IS NULL OR finisheddate > '%s')"%date
        else:
            t = "finisheddate IS NULL"

        if WHERE:
            WHERE = "%s AND %s"%(WHERE,t)
        else:
            WHERE = " WHERE %s"%t

    Sort = Properties['sort']
    if Sort:
        sort_attr = Sort['attribute']
        if sort_attr == 'date':
            sort_attr = Properties['LCdate']
        elif sort_attr == 'owners':
            sort_attr = 'owner1'

        WHERE = WHERE + " ORDER BY " + sort_attr
        #if not direction: WHERE = WHERE + " DESC"   works because ASC is the default
        if not Sort['direction']:
            WHERE = WHERE + " DESC" 

    sql = "SELECT priority,name,createdate,finisheddate,duedate,owner1,owner2,owner3,id,timestamp FROM %s %s"%(table,WHERE)

    try:
        cursor.execute(sql)
    except:
        g.pr("Cannot read %s: %s"%(Properties['host'],table))
        return None #[]
    else:
        return cursor.fetchall()



.. @+node:ekr.20161117164321.535: *11* CreateAndDisplayList (returns Item List)
def CreateAndDisplayList(self, results):
    LCtrl = self.ListCtrls[self.L]
    LCdate = self.PropertyDicts[self.L]['LCdate']
    if LCdate == 'timestamp':
        format = '%m/%d %H:%M:%S'
    else:
        format = '%m/%d/%y'
    itemlist = []

    LCtrl.DeleteAllItems()
    class Item: pass

    for x,row in enumerate(results):

        item = Item()
        << assign item attributes >>
        itemlist.append(item)

        << draw item >>

    return itemlist


.. @+node:ekr.20161117164321.536: *12* << assign item attributes >>
item.priority = int(row[0]) #int(row[0]) needs int because it seems to come back as a long from MySQL
item.name = row[1]
item.createdate = row[2]
item.finisheddate = row[3]
item.duedate = row[4]
item.owners = [y for y in row[5:8] if y] #if you carry around ['tom',None,None] Note this is 5:8 not 5:7
item.id = row[8]
item.timestamp = row[9]

.. @+node:ekr.20161117164321.537: *12* << draw item >>
LCtrl.InsertImageStringItem(x, str(item.priority), LCtrl.idx1)
LCtrl.SetStringItem(x,1,item.name)
LCtrl.SetStringItem(x,2,'; '.join(item.owners))
date = item.__dict__[LCdate]
LCtrl.SetStringItem(x,3,date and date.Format(format) or "")

if item.finisheddate:
    LC_Item = LCtrl.GetItem(x)
    LC_Item.SetImage(LCtrl.idx0) #might just want generic number or greyed one two three
    LC_Item.SetTextColour(wxLIGHT_GREY)
    LCtrl.SetItem(LC_Item)

elif item.priority==2:
    LC_Item = LCtrl.GetItem(x)
    f = self.LC_font
    f.SetWeight(wxBOLD)
    LC_Item.SetFont(f)
    f.SetWeight(wxNORMAL) #resetting weight
    LCtrl.SetItem(LC_Item)

elif item.priority==3:
    LC_Item = LCtrl.GetItem(x)
    f = self.LC_font
    f.SetWeight(wxBOLD)
    LC_Item.SetFont(f)
    f.SetWeight(wxNORMAL) #return to normal
    LC_Item.SetTextColour(wxRED)
    LCtrl.SetItem(LC_Item)
.. @+node:ekr.20161117164321.538: *11* OnSync
def OnSync(self, evt=None):
    if self.modified:
        self.OnUpdate()
    #Note that the results of an sqlite query are an instance that you need to turn into a tuple or MySQL gets unhappy

    if OFFLINE_ONLY:
        dlg = wxMessageDialog(self, "You need to be online to synchronize!", style = wxOK|wxICON_ERROR)
        dlg.ShowModal()
        dlg.Destroy()
        return

    dlg = wxMessageDialog(self,"Synchronize Table(s): "+" and ".join(SYNC_TABLES),"Synchronize...",wxICON_QUESTION|wxYES_NO)
    val = dlg.ShowModal()
    dlg.Destroy()
    if val == wxID_NO:
        return

    if REMOTE_HOST is None:
        g.pr("There doesn't appear to be a Remote Server")
        return

    if LOCAL_HOST is None:
        g.pr("There doesn't appear to be a Local Server")
        return

    g.pr("LOCAL_HOST=",LOCAL_HOST)
    g.pr("REMOTE_HOST=",REMOTE_HOST)

    r_cursor = self.GetCursor(REMOTE_HOST)
    if r_cursor is None:
        g.pr("Couldn't get a cursor for %s"%REMOTE_HOST)
        return

    l_cursor = self.GetCursor(LOCAL_HOST)
    if l_cursor is None:
        g.pr("Couldn't get a cursor for %s"%LOCAL_HOST)
        return

    # moving the sync time back a second to make sure that we don't lose track of any nodes
    #that are being updated or inserted at the same time as we are syncing
    r_cursor.execute("SELECT NOW()")
    l_now = mx.DateTime.now()-mx.DateTime.oneSecond
    r_now = r_cursor.fetchone()[0]-mx.DateTime.oneSecond
    #because of some inconsistent rounding appears necessary to make sure the sqlite timestamp is less than l_now
    #having seen same issue for mysql but for consistency (and because sqlite could also be "server" rdbms
    l_ts = l_now - mx.DateTime.DateTimeDelta(0,0,0,0.02)
    r_ts = r_now - mx.DateTime.DateTimeDelta(0,0,0,0.02)
    g.pr("l_now=",l_now, "l_ts =",l_ts)
    g.pr("r_now=",r_now, "r_ts=",r_ts)

    r_cursor.execute("SELECT MAX(last_sync) FROM user_sync WHERE user = %s", (USER,))
    r_last_sync = r_cursor.fetchone()[0]
    g.pr("last sync (remote time) =",r_last_sync)

    l_cursor.execute("SELECT MAX(last_sync) FROM user_sync")
    l_last_sync = l_cursor.fetchone()[0] #note MAX returns a string with sqlite so we turn it make into DateTime
    l_last_sync = mx.DateTime.DateTimeFrom(l_last_sync)
    g.pr("last sync (local time) =",l_last_sync)

    for table in SYNC_TABLES:
        # Need to pick up changes for both so syncing one doesn't add new things and screw up the second sync
        g.pr("Checking "+table+" on the Remote Server; changes (excluding deletes) are:")
        r_cursor.execute("SELECT id,createdate from "+table+" WHERE timestamp > %s AND timestamp <= %s",(r_last_sync,r_now)) 
        r_results = r_cursor.fetchall()
        g.pr("Server changes (excluding deletes)")
        g.pr(r_results)

        g.pr("Checking "+table+" on Local; changes (excluding deletes) are:")
        l_cursor.execute("SELECT id,createdate from "+table+" WHERE timestamp > %s AND timestamp <= %s",(l_last_sync,l_now))
        l_results = l_cursor.fetchall()
        g.pr("Local changes (excluding deletes)")
        g.pr(l_results)

        for id, createdate in r_results:
            r_cursor.execute("SELECT priority,name,owner1,owner2,owner3,createdate,finisheddate,duedate,note,id FROM "+table+" WHERE ID = %s",(id,))
            row = r_cursor.fetchone()
            if row:
                if createdate > r_last_sync:
                    l_cursor.execute("INSERT INTO "+table+" (priority,name,owner1,owner2,owner3,createdate,finisheddate,duedate,note,id) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)", row) #*row also works
                    g.pr("Created %s in %s on Local"%(id,table))
                else:
                    l_cursor.execute("UPDATE "+table+" SET priority = %s, name =%s, owner1 = %s, owner2 = %s, owner3 = %s, createdate = %s, finisheddate = %s, duedate = %s, note = %s WHERE id = %s", row)
                    g.pr("Updated %s in %s on Local"%(id,table))
                # for reasons I don't understand l_now here is a 1/100 ahead of l_now when inserted into user_sync
                l_cursor.execute("UPDATE "+table+" SET timestamp = %s WHERE id = %s", (l_ts,id))

        for id, createdate in l_results:
            l_cursor.execute("SELECT priority,name,owner1,owner2,owner3,createdate,finisheddate,duedate,note,id FROM "+table+" WHERE ID = %s",(id,))
            row = l_cursor.fetchone()
            if row:
                row = tuple(row)
                #above needed because sqlite returns an enhanced tuple-like object that is not a tuple
                if createdate > l_last_sync:
                    r_cursor.execute("INSERT INTO "+table+" (priority,name,owner1,owner2,owner3,createdate,finisheddate,duedate,note,id) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)", row)
                    g.pr("Created %s in %s on Server"%(id,table))
                else:
                    r_cursor.execute("UPDATE "+table+" SET priority = %s, name =%s, owner1 = %s, owner2 = %s, owner3 = %s, createdate = %s, finisheddate = %s, duedate = %s, note = %s WHERE id = %s", row)
                    g.pr("Updated %s in %s on Server"%(id,table))
                r_cursor.execute("UPDATE "+table+" SET timestamp = %s WHERE id = %s", (r_ts,id))

    #Handle the deletes; Note if at some point only 'd's are being written won't have to check for 'd'
    r_cursor.execute("SELECT id,table_name FROM sync WHERE timestamp > %s AND timestamp <= %s AND action = 'd'",(r_last_sync,r_now))
    r_results = r_cursor.fetchall()

    l_cursor.execute("SELECT id,table_name FROM sync WHERE timestamp > %s AND timestamp <= %s AND action = 'd'",(l_last_sync,l_now))
    l_results = l_cursor.fetchall()

    for id,table in l_results:
        r_cursor.execute("DELETE from "+table+" WHERE id = %s", (id,))
        g.pr("Deleted %s from %s on Server (if it existed there)"%(id,table))

    for id,table in r_results:
        l_cursor.execute("DELETE from "+table+" WHERE id = %s", (id,))
        g.pr("Deleted %s from %s on Local (if it existed there)"%(id,table)	)
    #End of deletes code

    #update the user_sync database with the latest sync times
    l_cursor.execute("INSERT INTO user_sync (user,last_sync) VALUES (%s,%s)", (USER,l_now)) #don't really need USER for local
    r_cursor.execute("INSERT INTO user_sync (user,last_sync) VALUES (%s,%s)", (USER,r_now)) 

    g.pr("Synchronization completed")

.. @+node:ekr.20161117164321.539: *11* TimeStamper
def TimeStamper(self, host, cursor, table, id):
    #note that you can insert a timestamp value into an mysql timestamp field
    if host.split(':')[1] == 'sqlite': #host -> location:rdbms
        timestamp = mx.DateTime.now()
        cursor.execute("UPDATE "+table+" SET timestamp = %s WHERE id = %s", (timestamp,id))
    else:
        cursor.execute("Select timestamp from "+table+" WHERE id = %s", (id,))
        timestamp = cursor.fetchone()[0]

    return timestamp
.. @+node:ekr.20161117164321.540: *10* Evaluate methods
.. @+node:ekr.20161117164321.541: *11* OnShowEvaluate
def OnShowEvaluate(self, evt=None):

    self.EvalDialog.Show(True)
    self.EvalDialog.EvalText.SetSelection(-1,-1)
    self.EvalDialog.EvalText.SetFocus()

.. @+node:ekr.20161117164321.542: *11* OnEvaluate
def OnEvaluate(self, evt=None):
    expr = self.EvalDialog.EvalText.GetValue()
    g.pr("%s => "%expr,newline=False)
    g.pr(eval(expr))

.. @+node:ekr.20161117164321.543: *10* Help menu methods
.. @+node:ekr.20161117164321.544: *11* OnShowAbout
def OnShowAbout(self, evt=None):
    from about import AboutBox
    dlg = AboutBox(self, app_version = VERSION)
    dlg.ShowModal()
    dlg.Destroy()

.. @+node:ekr.20161117164321.545: *11* OnShowHelp
def OnShowHelp(self, evt=None):
    os.startfile('ListManager.chm')

.. @+node:ekr.20161117164321.546: *10* GetUID
def GetUID(self):
    pyiid = CreateGuid()
    # the str(pyiid) looks like {....} and doing [1:-1] strips that off
    return str(pyiid)[1:-1]

.. @+node:ekr.20161117164321.547: *10* OnIdle
def OnIdle(self, evt):	
    << Check for Transfers From Outlook >>
    << Check if Edited File has Changed >>

.. @+node:ekr.20161117164321.548: *11* << Check for Transfers From Outlook >>
if OUTLOOK:
    input,output,exc = select.select([self.sock],[],[],0)
    if input:
        client,addr = self.sock.accept() # Get a connection
        rec = client.recv(8192)
        d = pickle.loads(rec)

        class Item: pass

        item = Item()
        item.id = self.GetUID()
        item.priority = 1
        item.createdate = mx.DateTime.now()
        item.duedate = item.finisheddate = None

        #outlook strings are unicode; ascii encode makes sure no chars above 127
        name = d['Subject'].encode('ascii','replace') 
        item.name = name[:150]

        owner = d['SenderName'].encode('ascii','replace') #encode takes unicode to standard strings
        owner = owner[:25]
        item.owners = [owner]

        note = d['CreationTime'] + '\n' + d['Body'].encode('ascii','replace')
        #foldername = d['Parent.Name']

        #location, rdbms, table = MAIL_LIST_PATH.split(':')
        #host = '%s:%s'%(location,rdbms)
        host, table = re.split('(.*?:.*?):', MAIL_LIST_PATH)[1:3] #really just for fun

        cursor = self.Cursors[host]

        cursor.execute("INSERT INTO "+table+" (priority,name,createdate,finisheddate,duedate,owner1,note,id) VALUES (%s,%s,%s,%s,%s,%s,%s,%s)",
            (item.priority, name, item.createdate, item.finisheddate, item.duedate, owner, note, item.id))

        item.timestamp = self.TimeStamper(host, cursor, table, item.id)

        #check to see if table is open
        for L,Properties in enumerate(self.PropertyDicts):
            if Properties['table'] == table and Properties['host'] == host:
                break
        else:
            g.pr("Table not open but wrote to database anyway") #Needs to be a dialog box
            return

        # could have started to edit something and never finished it
        if self.modified:
            self.OnUpdate()

        if self.L != L:
            self.nb.SetSelection(L) # Note that this does not call OnPageChange if the page doesn't change

        LCtrl = self.ListCtrls[L]

        if self.curIdx != -1:
            LCtrl.SetItemState(self.curIdx, 0, wxLIST_STATE_SELECTED)

        self.ItemLists[L].insert(0,item)    
        LCtrl.InsertImageStringItem(0,"1", LCtrl.idx1)
        LCtrl.SetStringItem(0,self.attr2col_num['name'],name)
        LCtrl.SetStringItem(0, self.attr2col_num['owners'], owner)

        if Properties['LCdate'] == 'timestamp':
            LCtrl.SetStringItem(0, self.attr2col_num['date'], item.timestamp.Format("%m/%d %H:%M:%S"))
        elif Properties['LCdate'] == 'createdate':
            LCtrl.SetStringItem(0, self.attr2col_num['date'], item.createdate.Format('%m/%d/%y'))

        LCtrl.SetItemState(0, wxLIST_STATE_SELECTED, wxLIST_STATE_SELECTED)
        self.curIdx = 0 

.. @+node:ekr.20161117164321.549: *11* << Check if Edited File has Changed >>
for ed in self.editor:
    path = ed['path']
    t = os.path.getmtime(path)
    if t != ed['time']:
        f = file(path,'r')
        note = f.read()
        f.close()
        ed['time'] = t

        host = ed['host']
        cursor = self.Cursors[host]
        table = ed['table']
        id = ed['id']
        cursor.execute("UPDATE "+table+" SET note = %s WHERE id = %s", (note,id)) 
        # see @rst documentation note
        ts = self.TimeStamper(host, cursor, table, id)

        idx = self.curIdx
        L = self.L
        if idx != -1:
            item = self.ItemLists[L][idx]
            if item.id == id:
                self.note.SetValue(note)
                item.timestamp = ts

                if self.PropertyDicts[L]['LCdate'] == 'timestamp':
                    self.ListCtrls[L].SetStringItem(idx, self.attr2col_num['date'], item.timestamp.Format("%m/%d %H:%M:%S"))

                if 'note' in self.modified: #if necessary only if somehow note text didn't change
                    del self.modified['note']

.. @+node:ekr.20161117164321.550: *9* class ListCtrl
class ListCtrl(wxListCtrl, wxListCtrlAutoWidthMixin):
    @others
.. @+node:ekr.20161117164321.551: *10* __init__
def __init__(self, parent, ID, pos=wxDefaultPosition, size=wxDefaultSize, style=0):
    wxListCtrl.__init__(self, parent, ID, pos, size, style)
    wxListCtrlAutoWidthMixin.__init__(self)

    self.il = wxImageList(16,16)

    sm_up = self.il.Add(wxBitmap('bitmaps\\up_arrow.bmp')) #(images.getSmallUpArrowBitmap())
    sm_dn = self.il.Add(wxBitmap('bitmaps\\down_arrow.bmp'))
    self.arrows = (sm_up,sm_dn)

    self.idx1 = self.il.Add(wxBitmap('bitmaps\\box.bmp'))
    self.idx0 = self.il.Add(wxBitmap('bitmaps\\filledwhitebox.bmp'))    

    self.SetImageList(self.il, wxIMAGE_LIST_SMALL)

    EVT_LIST_COL_BEGIN_DRAG(self, self.GetId(), self.OnColBeginDrag)    

    self.SetUpColumns()

.. @+node:ekr.20161117164321.552: *10* SetUpColumns
def SetUpColumns(self):
    #Need to to construct column heads for columns with sorting by hand to get sorting images on columns
    info = wxListItem()
    info.m_mask = wxLIST_MASK_TEXT | wxLIST_MASK_IMAGE | wxLIST_MASK_FORMAT
    info.m_image = -1

    #Oth column is priority which is sortable
    info.m_format = wxLIST_FORMAT_LEFT
    info.m_text = "P"
    self.InsertColumnInfo(0, info)
    self.SetColumnWidth(0, 35)

    self.InsertColumn(1, "Name")
    self.SetColumnWidth(1, 590)

    self.InsertColumn(2, "Owner")
    self.SetColumnWidth(2, 100)

    #3th column is create ate and same as with priority - needs to constructed by hand
    info.m_format = wxLIST_FORMAT_LEFT
    info.m_text = "Due Date"
    self.InsertColumnInfo(3, info)
    self.SetColumnWidth(3, 75)                

.. @+node:ekr.20161117164321.553: *10* OnColBeginDrag
def OnColBeginDrag(self, evt):
    #if inplace editor then change its dimensions
    if evt.GetColumn() == 0:
        evt.Veto()
.. @+node:ekr.20161117164321.554: *9* class MyApp
class MyApp(wxApp):
    @others
.. @+node:ekr.20161117164321.555: *10* OnInit
def OnInit(self):
    global OFFLINE_ONLY, CANCEL
    wxInitAllImageHandlers()

    if STARTUP_DIALOG:
        startup = StartupDialog(None, 'List Manager')
        val = startup.ShowModal()
        startup.Destroy()
        if val == wxID_YES:
            OFFLINE_ONLY = True
        elif val == wxID_NO:
            OFFLINE_ONLY = False
        elif val == wxID_CANCEL:
            CANCEL = True
            return True

    if OFFLINE_ONLY is False:
        server = REMOTE_HOST.split(':')[0]
        try:
            socket.gethostbyname(server)
        except:
            dlg = wxMessageDialog(None, "Cannot connect to remote server! Only offline access is possible.", "ListManager", style=wxOK|wxICON_EXCLAMATION|wxSTAY_ON_TOP)
            dlg.ShowModal()
            dlg.Destroy()
            OFFLINE_ONLY = True

    frame = ListManager(None, -1, "List Manager", size = (X,Y))
    frame.Show(True)
    self.SetTopWindow(frame)
    CANCEL = False
    return True


.. @+node:ekr.20161117164321.556: *9* class Logger
class Logger:
    def __init__(self):
        self.dlg = LoggerDialog(None, "", "Alerts and Exceptions", dir=DIRECTORY)
    def write(self, error_msg):
        if not self.dlg.IsShown():
            self.dlg.text.AppendText("\n%s\n"%time.asctime())
            self.dlg.Show(True)

        self.dlg.text.AppendText(error_msg)

.. @+node:ekr.20161117164321.557: *9* run
def run():
    app = MyApp(0)
    if not CANCEL:
        sys.stderr = sys.stdout = Logger()
        app.MainLoop()

if __name__ == '__main__':
    run()
.. @+node:ekr.20161117164321.558: *8* LMDialogs.py
@ @rst-options
code_mode = True
@c


@language python
from wxPython.wx import *
# the following two are needed for the calendar
from wxPython.calendar import *
from wxPython.utils import *
import os
@others
.. @+node:ekr.20161117164321.559: *9* class PopDialog
class TicklerDialog(wxDialog):
    @others
.. @+node:ekr.20161117164321.560: *10* __init__
def __init__(self, parent, msg, caption, pos = wxDefaultPosition, size = wxDefaultSize):
    wxDialog.__init__(self, parent, -1, caption, pos, size, style=wxSTAY_ON_TOP | wxTHICK_FRAME | wxCAPTION)

    TC = wxTextCtrl(self, -1, msg, wxDefaultPosition,
                    (450,250),
                    wxTE_MULTILINE | wxTE_READONLY | wxTE_RICH2)

    sizer = wxBoxSizer(wxVERTICAL)
    box = wxBoxSizer(wxHORIZONTAL)        

    sizer.Add(TC, 1, wxALIGN_CENTRE|wxALL, 5)
    line = wxStaticLine(self, -1, size = (20,-1), style = wxLI_HORIZONTAL)

    sizer.Add(line, 0, wxGROW|wxALIGN_CENTER_VERTICAL|wxTOP, 5)
    btn = wxButton(self, wxID_OK, "GO TO ITEM")
    box.Add(btn, 0, wxALIGN_CENTRE|wxALL, 5)
    btn.SetDefault()

    btn = wxButton(self, wxID_FORWARD, "SHOW NEXT")
    box.Add(btn, 0, wxALIGN_CENTRE|wxALL, 5)

    btn = wxButton(self, wxID_APPLY, "MAIL")
    box.Add(btn, 0, wxALIGN_CENTRE|wxALL, 5)        

    btn = wxButton(self, wxID_CANCEL, "CANCEL")
    box.Add(btn, 0, wxALIGN_CENTRE|wxALL, 5)

    sizer.AddSizer(box, 0, wxGROW|wxALIGN_CENTER_VERTICAL|wxALL, 5)
    self.SetSizer(sizer)
    self.SetAutoLayout(True)
    sizer.Fit(self)

    EVT_LEFT_DOWN(TC, self.OnLeftDown)
    EVT_BUTTON(self, wxID_FORWARD, self.OnForward)
    EVT_BUTTON(self, wxID_APPLY, self.OnMail)

    TC.SetCursor(wxStockCursor(wxCURSOR_ARROW))        

    self.TC = TC
.. @+node:ekr.20161117164321.561: *10* OnLeftDown
def OnLeftDown(self, evt):
    self.EndModal(wxID_OK)
.. @+node:ekr.20161117164321.562: *10* OnForward
def OnForward(self, evt):
    self.EndModal(wxID_FORWARD)
.. @+node:ekr.20161117164321.563: *10* OnMail
def OnMail(self, evt):
    self.EndModal(wxID_APPLY)        
.. @+node:ekr.20161117164321.564: *9* class StartupDialog
class StartupDialog(wxDialog):
    @others
.. @+node:ekr.20161117164321.565: *10* __init__
def __init__(self, parent, caption, pos=wxDefaultPosition, size=(300,115)):
    wxDialog.__init__(self, parent, -1, caption, pos, size, style=wxSTAY_ON_TOP|wxCAPTION)

    msg = "You can connect to the server using the network,\nor work offline, or cancel this logon."

    image = wxStaticBitmap(self, -1, wxBitmap('bitmaps\\wxpdemo.bmp'), (-1,-1), size=(32,32)) #sizer determines position
    text = wxStaticText(self, -1, msg, (-1,-1), size=(250,32)) #sizer determines position

    rect = wxBoxSizer(wxHORIZONTAL)
    rect.Add(image, 0, wxALIGN_LEFT|wxALL, 4)
    rect.Add(text, 1, wxALIGN_CENTER|wxTOP, 7)
    sizer = wxBoxSizer(wxVERTICAL)


    box = wxBoxSizer(wxHORIZONTAL)
    btn = wxButton(self, wxID_NO, 'Connect')
    box.Add(btn, 0, wxALL, 10)
    btn.SetDefault()

    btn = wxButton(self, wxID_YES, 'Work Offline')
    box.Add(btn, 0, wxALL, 10)

    btn = wxButton(self, wxID_CANCEL, 'Cancel')
    box.Add(btn, 0, wxALL, 10)

    sizer.AddSizer(rect)
    sizer.AddSizer(box)

    self.SetSizer(sizer)

    EVT_BUTTON(self, wxID_NO, self.OnSelection)
    EVT_BUTTON(self, wxID_YES, self.OnSelection)
    EVT_BUTTON(self, wxID_CANCEL, self.OnSelection)
.. @+node:ekr.20161117164321.566: *10* OnSelection
def OnSelection(self,evt):
    val = evt.GetId()
    self.EndModal(val)
.. @+node:ekr.20161117164321.567: *9* class ModifierDialog
class ModifierDialog(wxDialog):
    @others
.. @+node:ekr.20161117164321.568: *10* __init__
def __init__(self, parent, title,
             pos=wxDefaultPosition,
             size=wxDefaultSize,
             style=wxCAPTION,
             modifierlist=None,
             curselections = ''):
    wxDialog.__init__(self, parent, -1, title, pos, size, style)

    sizer1 = wxBoxSizer(wxVERTICAL)
    sizer2 = wxBoxSizer(wxHORIZONTAL)

    tc = wxTextCtrl(self, -1, "", size = (150,-1))
    sizer1.Add(tc, 0, wxALIGN_CENTRE|wxALL, 5)
    self.tc = tc

    if not modifierlist:
        modifierlist = []
    lb = wxListBox(self, -1,  wxDefaultPosition, (150,300), #wxPoint(90, 80)
                    modifierlist, wxLB_MULTIPLE|wxLB_SORT)

    sizer1.Add(lb, 1, wxALIGN_CENTRE|wxALL, 5)

    line = wxStaticLine(self, -1, size = (20,-1), style = wxLI_HORIZONTAL)
    sizer1.Add(line, 0, wxGROW|wxALIGN_CENTER_VERTICAL|wxTOP, 5)


    btn = wxButton(self, wxID_OK, "OK")
    sizer2.Add(btn, 0, wxALIGN_CENTRE|wxALL, 5)
    btn.SetDefault()

    btn = wxButton(self, wxID_CANCEL, "CANCEL")
    sizer2.Add(btn, 0, wxALIGN_CENTRE|wxALL, 5)

    sizer1.AddSizer(sizer2, 0, wxGROW|wxALIGN_CENTER_VERTICAL|wxALL, 5)
    self.SetSizer(sizer1)
    self.SetAutoLayout(True)
    sizer1.Fit(self)

    for sel in curselections:
        index = lb.FindString(sel)
        if index !=-1:
            lb.SetSelection(index)

    self.lb = lb

    EVT_BUTTON(self, wxID_CANCEL, self.ClearSelections)


.. @+node:ekr.20161117164321.569: *10* GetUserInput
def GetUserInput(self):
    idx_list = self.lb.GetSelections()
    mod_list =[]
    for i in idx_list:
        mod_list.append(self.lb.GetString(i))
        self.lb.Deselect(i) #071203

    new_list = []
    manual_string = self.tc.GetValue() #text entry box

    if manual_string:
        manual_list = [x.strip() for x in manual_string.split(';')]
        for name in manual_list:
            clean_name = ", ".join([x.strip().title() for x in name.split(',')])
            if clean_name not in mod_list:
                mod_list.append(clean_name)
                new_list.append(clean_name)


    return (mod_list, new_list)
.. @+node:ekr.20161117164321.570: *10* SelectCurrent
def SelectCurrent(self, cur_sel):
    for sel in cur_sel:
        index = self.lb.FindString(sel)
        if index !=-1:
            self.lb.SetSelection(index)


.. @+node:ekr.20161117164321.571: *10* ClearSelections
def ClearSelections(self, evt=None):
    idx_list = self.lb.GetSelections() #note you can't just use the indexes of the SelectCurrent since they may have clicked before cancelling
    for i in idx_list:
        self.lb.Deselect(i)

    evt.Skip()
.. @+node:ekr.20161117164321.572: *9* class MailDialog
class MailDialog(wxDialog):
    @others
.. @+node:ekr.20161117164321.573: *10* __init__
def __init__(self, parent, title,
             pos=wxDefaultPosition,
             size=wxDefaultSize,
             style=wxSTAY_ON_TOP| wxTHICK_FRAME|wxCAPTION|wxSYSTEM_MENU,
             recipients='',
             subject = '',
             body = ''):

    wxDialog.__init__(self, parent, -1, title, pos, size, style)

    sizer = wxBoxSizer(wxVERTICAL)
    box = wxBoxSizer(wxHORIZONTAL)

    recipients = "; ".join(recipients)
    label = wxStaticText(self, -1, "To:",wxDefaultPosition, size=(40,-1), style=wxALIGN_LEFT)
    RTC = wxTextCtrl(self, -1, recipients, size = (480,-1))
    box.Add(label)
    box.Add(RTC)

    #sizer.Add(10,10,0)      

    sizer.AddSizer(box)        

    box = wxBoxSizer(wxHORIZONTAL)       
    label = wxStaticText(self, -1, "Subject:",wxDefaultPosition, size=(40,-1),style=wxALIGN_LEFT)
    STC = wxTextCtrl(self, -1, subject, size = (480,-1)) 
    box.Add(label)
    box.Add(STC)

    sizer.AddSizer(box)
    sizer.Add(1, 5, 0)

    BTC = wxTextCtrl(self, -1, body, wxDefaultPosition, size = (500,400), style=wxTE_MULTILINE|wxTE_RICH2)

    sizer.Add(BTC)

    box = wxBoxSizer(wxHORIZONTAL)
    btn = wxButton(self, wxID_OK, "SEND MAIL")
    box.Add(btn, 0, wxALIGN_CENTRE|wxALL, 5)
    btn.SetDefault()

    btn = wxButton(self, wxID_CANCEL, "CANCEL")
    box.Add(btn, 0, wxALIGN_CENTRE|wxALL, 5)

    sizer.AddSizer(box)
    self.SetSizer(sizer)
    self.SetAutoLayout(True)
    sizer.Fit(self)

    self.RTC = RTC
    self.STC = STC
    self.BTC = BTC

.. @+node:ekr.20161117164321.574: *9* class CalendarDialog
class CalendarDialog(wxDialog):
    @others
.. @+node:ekr.20161117164321.575: *10* __init__
def __init__(self, parent, title,
             pos=wxDefaultPosition,
             size=wxDefaultSize,
             style=wxCAPTION,
             date=0):

    wxDialog.__init__(self, parent, -1, title, pos, size, style)

    if not date:
        date = wxDateTime_Now()

    cal = wxCalendarCtrl(self, -1, date, #pos = (25,50),
                         style = wxCAL_SHOW_HOLIDAYS | wxCAL_SUNDAY_FIRST)

    EVT_CALENDAR(self, cal.GetId(), self.OnCalSelected)

    #EVT_CLOSE(self, self.OnCloseWindow)          

    self.cal = cal

    # Set up control to display a set of holidays:
    EVT_CALENDAR_MONTH(self, cal.GetId(), self.OnChangeMonth)

    self.holidays = [(1,1), (10,31), (12,25) ]    # (these don't move around)

    self.OnChangeMonth()        

#-------------------------------------------------------------------------        
    sizer1 = wxBoxSizer(wxVERTICAL)
    sizer2 = wxBoxSizer(wxHORIZONTAL)

    sizer1.Add(cal, 0, wxALIGN_CENTRE|wxALL, 5)

    line = wxStaticLine(self, -1, size = (20,-1), style = wxLI_HORIZONTAL)
    sizer1.Add(line, 0, wxGROW|wxALIGN_CENTER_VERTICAL|wxRIGHT|wxTOP, 5)


    btn = wxButton(self, wxID_OK, "OK")
    btn.SetDefault()
    sizer2.Add(btn, 0, wxALIGN_CENTRE|wxALL, 5)

    btn = wxButton(self, wxID_CANCEL, "CANCEL")
    #btn.SetDefault()
    sizer2.Add(btn, 0, wxALIGN_CENTRE|wxALL, 5)

    sizer1.AddSizer(sizer2, 0, wxGROW|wxALIGN_CENTER_VERTICAL|wxALL, 5)
    self.SetSizer(sizer1)
    self.SetAutoLayout(True)
    sizer1.Fit(self)
.. @+node:ekr.20161117164321.576: *10* OnCalSelected
def OnCalSelected(self, evt):
    self.result = evt.GetDate()
    self.EndModal(wxID_OK)
.. @+node:ekr.20161117164321.577: *10* OnChangeMonth
def OnChangeMonth(self, evt=None):
    cur_month = self.cal.GetDate().GetMonth() + 1   # convert wxDateTime 0-11 => 1-12
    for month, day in self.holidays:
        if month == cur_month:
            self.cal.SetHoliday(day)        
.. @+node:ekr.20161117164321.578: *10* OnCloseWindow
def OnCloseWindow(self, event):
    #self.cal.Destroy
    #self.Destroy()
    g.pr("I got to close window")
.. @+node:ekr.20161117164321.579: *10* GetDate
def GetDate(self):
    return self.result
.. @+node:ekr.20161117164321.580: *9* class FindDialog
class FindDialog(wxDialog):
    @others
.. @+node:ekr.20161117164321.581: *10* __init__
def __init__(self, parent, caption, msg, pos=wxDefaultPosition, size=(300,120)):
    wxDialog.__init__(self, parent, -1, caption, pos, size, style=wxSTAY_ON_TOP|wxCAPTION)    

    self.FindText = wxTextCtrl(self, -1, msg, wxDefaultPosition,(200,24))

    box_a = wxBoxSizer(wxHORIZONTAL)
    box_a.Add(self.FindText, 1, wxALIGN_CENTER|wxALL, 5)

    box_b = wxBoxSizer(wxVERTICAL)        
    btn = wxButton(self, wxID_OK, "OK")
    box_b.Add(btn, 0, wxALIGN_CENTER|wxALL,5)
    btn.SetDefault()               

    btn = wxButton(self, wxID_CANCEL, "CANCEL")
    box_b.Add(btn, 0, wxALIGN_CENTER)

    box_a.AddSizer(box_b)

    self.MatchCase = wxCheckBox(self, -1, "Match Case")
    self.MatchWhole = wxCheckBox(self, -1, "Match Whole Word")
    box_c = wxBoxSizer(wxVERTICAL)
    box_c.Add(self.MatchCase, 0, wxLEFT|wxBOTTOM, 5)
    box_c.Add(self.MatchWhole, 0, wxLEFT, 5)

    self.SearchNotes = wxCheckBox(self, -1, "Search Notes")
    self.SearchFinished = wxCheckBox(self, -1, "Search Finished")
    box_d = wxBoxSizer(wxVERTICAL)
    box_d.Add(self.SearchNotes, 0, wxLEFT|wxBOTTOM, 5)
    box_d.Add(self.SearchFinished, 0, wxLEFT, 5)

    box_e = wxBoxSizer(wxHORIZONTAL)
    box_e.AddSizer(box_c)
    box_e.AddSizer(box_d)

    sizer = wxBoxSizer(wxVERTICAL)
    sizer.AddSizer(box_a)
    sizer.AddSizer(box_e)

    self.SetSizer(sizer)

    EVT_BUTTON(self, wxID_OK, parent.FindString)


.. @+node:ekr.20161117164321.582: *9* class EvalDialog
class EvalDialog(wxDialog):
    @others
.. @+node:ekr.20161117164321.583: *10* __init__
def __init__(self, parent, caption, msg, pos=wxDefaultPosition, size=(300,80)):
    wxDialog.__init__(self, parent, -1, caption, pos, size, style=wxSTAY_ON_TOP|wxCAPTION)    

    EvalText = wxTextCtrl(self, -1, msg, wxDefaultPosition,(200,24))


    box_a = wxBoxSizer(wxHORIZONTAL)
    box_a.Add(EvalText, 1, wxALIGN_CENTER|wxALL, 5)

    box_b = wxBoxSizer(wxVERTICAL)        
    btn = wxButton(self, wxID_OK, "OK")
    box_b.Add(btn, 0, wxALIGN_CENTER|wxALL,5)
    btn.SetDefault()               

    btn = wxButton(self, wxID_CANCEL, "CANCEL")
    box_b.Add(btn, 0, wxALIGN_CENTER)

    box_a.AddSizer(box_b)

    self.SetSizer(box_a)

    self.EvalText = EvalText
    self.parent = parent

    #EVT_BUTTON(self, wxID_OK, self.PostOKEvent)
    EVT_BUTTON(self, wxID_OK, parent.OnEvaluate)



.. @+node:ekr.20161117164321.584: *10* PostOKEvent
def PostOKEvent(self, evt=None):
    wxPostEvent(self.parent, evt)
.. @+node:ekr.20161117164321.585: *9* class LoggerDialog
class LoggerDialog(wxDialog):
    @others
.. @+node:ekr.20161117164321.586: *10* __init__
def __init__(self, parent, msg, caption, pos=(-1,-1), size=(500,300), dir=None):
    wxDialog.__init__(self, parent, -1, caption, pos, size)
    #if pos == (-1,-1):
        #self.CenterOnScreen(wxBOTH)

    if dir:
        self.dir = dir
    else:
        self.dir = os.getcwd()

    text = wxTextCtrl(self, -1, msg, (-1,-1), (450,250), wxTE_MULTILINE | wxTE_READONLY)

    sizer = wxBoxSizer(wxVERTICAL)
    box = wxBoxSizer(wxHORIZONTAL)        

    sizer.Add(text, 1, wxALIGN_CENTRE|wxALL, 5)

    btn = wxButton(self, wxID_OK, "Close")
    box.Add(btn, 0, wxALIGN_CENTRE|wxALL, 5)
    btn.SetDefault()

    ID_SAVE = wxNewId()

    btn = wxButton(self, ID_SAVE, "Save to File")
    box.Add(btn, 0, wxALIGN_CENTRE|wxALL, 5)        

    sizer.AddSizer(box, 0, wxGROW|wxALIGN_CENTER_VERTICAL|wxALL, 5)
    self.SetSizer(sizer)
    self.SetAutoLayout(True)
    sizer.Fit(self)

    self.text = text

    EVT_BUTTON(self, ID_SAVE, self.OnSave)
.. @+node:ekr.20161117164321.587: *10* OnSave
def OnSave(self, evt):

    path = os.path.join(self.dir, 'logfile.txt')

    f = file(path,'a')
    f.write(self.text.GetValue())
    f.close()

    dlg = wxMessageDialog(self,"Appended text to logfile.text", "Notice", wxICON_INFORMATION|wxOK)
    dlg.ShowModal()
    dlg.Destroy()

    self.text.Clear()
.. @+node:ekr.20161117164321.588: *9* class FinishedDialog
class FinishedDialog(wxDialog):
    @others
.. @+node:ekr.20161117164321.589: *10* __init__
def __init__(self, parent, title,
            pos=wxDefaultPosition,
            size=wxDefaultSize,
            style=wxCAPTION,
            days=0,
            spin_label="",
            check_label=""):

    wxDialog.__init__(self, parent, -1, title, pos, size)
    self.Centre()

    self.check = wxCheckBox(self, -1, check_label)

    if days == -1:
        self.check.SetValue(True)
        days = 0

    panel = wxPanel(self, -1, (-1,-1),(225,75))
    wxStaticText(panel, -1, spin_label,(15, 15))
    self.text = wxTextCtrl(panel, -1, str(days), (30, 50), (30, -1))
    h = self.text.GetSize().height
    self.spin = wxSpinButton(panel, -1, (56, 50), (h, h), wxSP_VERTICAL)
    wxStaticText(panel, -1, 'days',(76, 53))
    self.spin.SetRange(0, 14)
    self.spin.SetValue(days)

    H_sizer = wxBoxSizer(wxHORIZONTAL)

    line = wxStaticLine(self, -1, size = (20,-1), style = wxLI_HORIZONTAL)

    btn = wxButton(self, wxID_OK, "OK")
    H_sizer.Add(btn, 0, wxALIGN_CENTRE|wxALL, 5)
    btn.SetDefault()

    btn = wxButton(self, wxID_CANCEL, "CANCEL")
    H_sizer.Add(btn, 0, wxALIGN_CENTRE|wxALL, 5)

    V_sizer = wxBoxSizer(wxVERTICAL)
    V_sizer.Add(panel,1,wxALIGN_CENTER|wxEXPAND)
    V_sizer.Add(-1,5)
    V_sizer.Add(self.check,0,wxALIGN_LEFT|wxALL,5)
    V_sizer.Add(line,0, wxGROW|wxALIGN_CENTER_VERTICAL|wxTOP, 5)
    V_sizer.AddSizer(H_sizer, 0, wxGROW|wxALIGN_CENTER_VERTICAL|wxALL, 5)

    self.SetSizer(V_sizer)
    self.SetAutoLayout(True)
    V_sizer.Fit(self)

    EVT_SPIN(self, self.spin.GetId(), self.OnSpin)
    EVT_CHECKBOX(self, self.check.GetId(), self.OnCheck)

    if self.check.GetValue():
        self.spin.Enable(False)
        self.text.Enable(False)

    self.Layout() #doesn't appear necessary


.. @+node:ekr.20161117164321.590: *10* OnSpin
def OnSpin(self, evt):
    self.text.SetValue(str(evt.GetPosition()))
.. @+node:ekr.20161117164321.591: *10* OnCheck
def OnCheck(self, evt=None):
    if self.check.GetValue():
        self.spin.Enable(False)
        self.text.Enable(False)
    else:
        self.spin.Enable(True)
        self.text.Enable(True)
.. @+node:ekr.20161117164321.592: *9* class TreeDialog
class TreeDialog(wxDialog):
    @others
.. @+node:ekr.20161117164321.593: *10* __init__
def __init__(self, parent, caption, pos=wxDefaultPosition, size=(300,400), tree={}):
    wxDialog.__init__(self, parent, -1, caption, pos, size, style=wxSTAY_ON_TOP|wxCAPTION)

    TreeCtrl = wxTreeCtrl(self, -1, wxDefaultPosition, (300,400), wxTR_HAS_BUTTONS)    #|wxTR_HIDE_ROOT)#wxDefaultSize,

    sizer = wxBoxSizer(wxVERTICAL)
    sizer.Add(TreeCtrl, 1, wxALIGN_CENTER|wxALL, 5)

    box = wxBoxSizer(wxHORIZONTAL)
    btn = wxButton(self, wxID_OK, "OK")
    box.Add(btn, 0, wxALIGN_CENTRE|wxALL, 5)
    btn.SetDefault()

    btn = wxButton(self, wxID_CANCEL, "CANCEL")
    box.Add(btn, 0, wxALIGN_CENTRE|wxALL, 5)

    sizer.AddSizer(box)
    self.SetAutoLayout(1)
    self.SetSizer(sizer)

    il = wxImageList(16,16)

    fldridx = il.Add(wxBitmap('bitmaps\\folder.bmp'))
    fldropenidx = il.Add(wxBitmap('bitmaps\\folder_open.bmp'))
    listidx =  il.Add(wxBitmap('bitmaps\\list.bmp'))

    TreeCtrl.SetImageList(il)

    root = TreeCtrl.AddRoot("List Manager")
    TreeCtrl.SetItemImage(root, fldridx, wxTreeItemIcon_Normal)
    TreeCtrl.SetItemImage(root, fldropenidx, wxTreeItemIcon_Expanded)

    for host in tree:
        child = TreeCtrl.AppendItem(root, host)
        TreeCtrl.SetItemImage(child, fldridx, wxTreeItemIcon_Normal)
        TreeCtrl.SetItemImage(child, fldropenidx, wxTreeItemIcon_Expanded)
        for listname in tree[host]:
            last = TreeCtrl.AppendItem(child, listname)
            TreeCtrl.SetItemImage(last, listidx, wxTreeItemIcon_Normal)
            TreeCtrl.SetItemImage(last, listidx, wxTreeItemIcon_Selected)

    TreeCtrl.Expand(root)

    self.TreeCtrl= TreeCtrl
    self.il = il #? prevents GC

    EVT_LEFT_DCLICK(TreeCtrl, self.OnLeftDClick)
.. @+node:ekr.20161117164321.594: *10* OnLeftDClick:
def OnLeftDClick(self, event=None):
    self.EndModal(wxID_OK)
.. @+node:ekr.20161117164321.595: *8* outlookAddin.py
@ @rst-options
code_mode = True
@c

@language python
<< outlookAddin declarations >>
@others

if __name__ == '__main__':
    import win32com.server.register
    win32com.server.register.UseCommandLine(OutlookAddin)
    if "--unregister" in sys.argv:
        UnregisterAddin(OutlookAddin)
    else:
        RegisterAddin(OutlookAddin)
.. @+node:ekr.20161117164321.596: *9* << outlookAddin declarations >>
# This is mainly stolen from Mark Hammond's demo plugin for win32com.client
# A demo plugin for Microsoft Outlook (NOT Outlook Express)

from win32com import universal
from win32com.server.exception import COMException
from win32com.client import gencache, DispatchWithEvents
from win32com.client import Dispatch
import winerror
import pythoncom
from win32com.client import constants
import win32ui ##
import sys
from socket import *
import pickle

# Support for COM objects we use.
#sz comment gencache.EnsureModule makes sure you are using makepy if the makepy-derived
#file doesn't already exist
#but as long as you did run makepy then you should just be alble to do a normal dispatch

mod = gencache.EnsureModule('{00062FFF-0000-0000-C000-000000000046}', 0, 9, 0, bForDemand=True) # Outlook 9
gencache.EnsureModule('{2DF8D04C-5BFA-101B-BDE5-00AA0044DE52}', 0, 2, 1, bForDemand=True) # Office 9

# The TLB defining the interfaces we implement
universal.RegisterInterfaces('{AC0714F2-3D04-11D1-AE7D-00A0C90F26F4}', 0, 1, 0, ["_IDTExtensibility2"])

Target = 'mail_transfer'


.. @+node:ekr.20161117164321.597: *9* class ButtonEvent
class ButtonEvent:
    @others
.. @+node:ekr.20161117164321.598: *10* OnClick
def OnClick(self, button, cancel):
    #activeExplorer and MailTransferFolder are globals defined in OnConnection
    sel = activeExplorer.Selection

    for i in range(1,sel.Count+1):
        item = sel.Item(i)
        item.Move(MailTransferFolder)

    return cancel

.. @+node:ekr.20161117164321.599: *9* class FolderEvent
class FolderEvent:
    @others
.. @+node:ekr.20161117164321.600: *10* OnItemAdd
def OnItemAdd(self, item):
    try:
        s = socket(AF_INET,SOCK_STREAM)
        s.connect(('localhost', 8888))
        d = {}
        d['Parent.Name'] = item.Parent.Name
        d['SenderName'] = item.SenderName
        d['Subject'] = item.Subject
        d['Body'] = item.Body[:5000]
        d['CreationTime'] = item.CreationTime.Format()
        str = pickle.dumps(d)
        s.send(str) # ?Receive no more than 1024 bytes
        s.close()
        win32ui.MessageBox("Sent %s to ListManager"%item.Subject)
    except:
        pass
.. @+node:ekr.20161117164321.601: *9* class OutlookAddin
class OutlookAddin:
    << class OutlookAddin declarations >>
    @others
.. @+node:ekr.20161117164321.602: *10* << class OutlookAddin declarations >>
_com_interfaces_ = ['_IDTExtensibility2']
_public_methods_ = []
_reg_clsctx_ = pythoncom.CLSCTX_INPROC_SERVER
_reg_clsid_ = "{0F47D9F3-598B-4d24-B7E3-92AC15ED27E2}"
_reg_progid_ = "Python.Test.OutlookAddin"
_reg_policy_spec_ = "win32com.server.policy.EventHandlerPolicy"
.. @+node:ekr.20161117164321.603: *10* OnConnection
def OnConnection(self, application, connectMode, addin, custom):
    global MailTransferFolder
    global activeExplorer
    # ActiveExplorer may be none when started without a UI (eg, WinCE synchronisation)
    activeExplorer = application.ActiveExplorer()
    if activeExplorer:
        bars = activeExplorer.CommandBars
        toolbar = bars.Item("Standard")
        item = toolbar.Controls.Add(Type=constants.msoControlButton, Temporary=True)
        item = self.toolbarButton = DispatchWithEvents(item, ButtonEvent) #? just need this to be an ivar
        item.Caption="List Manager"
        item.TooltipText = "Click to move"
        item.Enabled = True
        #self.toolbarButton = DispatchWithEvents(item, ButtonEvent) #need something that won't get GC'd. Note Dispatch returns item

    ns = application.GetNamespace("MAPI")
    Folders = ns.Folders

    for i in range(1,len(Folders)+1):
        if Folders[i].Name.find("Mailbox") != -1:
            folders = Folders[i].Folders
            break
    else:
        win32ui.MessageBox("Can't find Mailbox!")
        return	

    for i in range(1,len(folders)+1):
        if folders[i].Name == Target:
            MailTransferFolder = folders[i]
            self.targetMailbox = DispatchWithEvents(folders[i].Items, FolderEvent) #? just need this to be an ivar
            win32ui.MessageBox("Enabled: %s\nOutlookAddin3"%Target)
            break
    else:
        win32ui.MessageBox("Could not find mail folder: %s\nOutlookAddin3"%Target)
.. @+node:ekr.20161117164321.604: *10* OnDisconnection
def OnDisconnection(self, mode, custom):
    g.pr("OnDisconnection")
.. @+node:ekr.20161117164321.605: *10* OnAddInsUpdate
def OnAddInsUpdate(self, custom):
    g.pr("OnAddInsUpdate", custom)
.. @+node:ekr.20161117164321.606: *10* OnStartupComplete
def OnStartupComplete(self, custom):
    g.pr("OnStartupComplete", custom)
.. @+node:ekr.20161117164321.607: *10* OnBeginShutdown
def OnBeginShutdown(self, custom):
    g.pr("OnBeginShutdown", custom)
.. @+node:ekr.20161117164321.608: *9* RegisterAddin
def RegisterAddin(klass):
    import _winreg
    key = _winreg.CreateKey(_winreg.HKEY_CURRENT_USER, "Software\\Microsoft\\Office\\Outlook\\Addins")
    subkey = _winreg.CreateKey(key, klass._reg_progid_)
    _winreg.SetValueEx(subkey, "CommandLineSafe", 0, _winreg.REG_DWORD, 0)
    _winreg.SetValueEx(subkey, "LoadBehavior", 0, _winreg.REG_DWORD, 3)
    _winreg.SetValueEx(subkey, "Description", 0, _winreg.REG_SZ, klass._reg_progid_)
    _winreg.SetValueEx(subkey, "FriendlyName", 0, _winreg.REG_SZ, klass._reg_progid_)
.. @+node:ekr.20161117164321.609: *9* UnregisterAddin
def UnregisterAddin(klass):
    import _winreg
    try:
        _winreg.DeleteKey(_winreg.HKEY_CURRENT_USER, "Software\\Microsoft\\Office\\Outlook\\Addins\\" + klass._reg_progid_)
    except WindowsError:
        pass
.. @+node:ekr.20161117164321.610: *7* @rst ../doc/pdfTest.pdf
########
Headline
########

@ @rst-options
.. These options have NO EFFECT for rst2 plugin!
code_mode=False
generate_rst=True
http_server_support = False
show_organizer_nodes=True
show_headlines=True
show_leo_directives=True
stylesheet_path=c:\prog\leoCVS\leo\doc
write_intermediate_file = True
verbose=True
@c

This is a test of pdf stuff

.. contents::
.. @+node:ekr.20161117164321.611: *8* child node
.. @+node:ekr.20161117164321.612: *9* @rst
child node text
.. @+node:ekr.20161117164321.613: *6* Tests of settings when opened from another .leo file
# g.app.config.updateSettings(c)
g.es('test_setting',c.config.getBool('test_setting'))
.. @+node:ekr.20161117164321.614: *6* Tk bindtags test
import Tkinter as Tk

root = Tk.Tk()
c = Tk.Canvas(root,background='white')
g.pr(c.bindtags())

if 0:
    c.pack(expand=1,fill='both')
    f = Tk.Frame(c)
    c.create_window(0,0,window=f,anchor='nw')
    f.pack_configure(fill='both',expand=1)
    body = olCreateControl(self,frame,f)
    c.on = False 
    sel = lambda event, c = c, body = body:select(event,c,body)
    ai = lambda event, c = c, body = body, colorizer = frame.body:add_item(event,c,body,colorizer.getColorizer())
    c.bind("<Key>",watcher,'+')
    c.bind("<Key>",sel,'+')
    c.bind("<Key>",ai,'+')
    ctags = c.bindtags()
    btags = body.bindtags()
    btags =(ctags[0],btags[0],btags[1],btags[2],btags[3])
    body.bindtags(btags)
.. @+node:ekr.20161117164321.615: *6* Unicode stuff
stuff = g.toEncodedString(u'∑','utf-8')
g.pr(type(stuff))
g.pr('*' * 10)
for ch in stuff:
    g.pr(ch, ord(ch),newline=False)
g.pr()
.. @+node:ekr.20161117164321.616: *6* Write to log tab
c.frame.log.selectTab('Log')
g.es('Test',color='blue')
.. @+node:ekr.20161117164321.617: *6* Write to test tab
c.frame.log.selectTab('Test')
g.es('Test',color='red',tabName='Test')
.. @+node:ekr.20161117164321.618: *5* Print a unicode character
@first # -*- coding: utf-8 -*-

s = g.ue('炰','utf-8')
g.es(s)
g.pr(s)
s = '炰'
g.pr(s)
.. @+node:ekr.20161117164321.619: *5* runProfile button mini-test
for i in range(10000):
    if i and (i % 1000) == 0:
        g.pr(i)
.. @+node:ekr.20161117164321.620: *5* runTimeit mini-test
i = 0
for i in range(100000):
    i += 1
    i -= 1
.. @+node:ekr.20161117164321.621: *5* Standard print test
@first # -*- coding: utf-8 -*-

import sys

print('=' * 40)

e = sys.getdefaultencoding()
print('encoding',e)

table = (
    'La Peña',
    g.u('La Peña'),
    # u'La Peña',
    g.u('La Pe\xf1a')
)

for s in table:
    print(type(s))
    g.es_print('g.es_print',s)
    if type(s) != type(u'a'):
        s = unicode(s,e)
    print('print     ',s)
    print('repr(s)   ',repr(s))
.. @+node:ekr.20161117164321.622: *5* User Icon tests
.. @+node:ekr.20161117164321.623: *6* Delete user icons
for p in c.allNodes_iter():

    if hasattr(p.v.t,"unknownAttributes"):
        a = p.v.t.unknownAttributes
        iconsList = a.get("icons")
        if dict:
            a["icons"] = []
            a["lineYOffset"] = 0

c.redraw()
.. @+node:ekr.20161117164321.624: *6* Test of user icons
p.v.t.unknownAttributes = {}
a = p.v.t.unknownAttributes

<< define event callbacks >>

path = g.os_path_join(g.app.loadDir,"..","Icons")
icon1 = g.os_path_join(path,"lt_arrow_enabled.gif")
icon2 = g.os_path_join(path,"rt_arrow_enabled.gif")

d1 = {
    "type" : "file", "file" : icon1,
    "where" : "beforeIcon",
    "yoffset" : -3,
    # "yoffset" : 5, "ypad" : -5,
    # "height" : 40, # automatically adjust headline y position.
    "xpad": 2
}

# Classes and functions can only be pickled if they are at the top level of a module.
    #"onClick" : onClick,
    #"onRightClick" : onRightClick,
    #"onDoubleClick" : onDoubleClick }

d2 = {
    "type" : "file", "file" : icon2,
    "where" : "beforeHeadline",
    "yoffset" : -3,
    "xoffset" : 2, "xpad" : -2 }

a["icons"] = [d1,d2] # [d1,d2]
a["lineYOffset"] = 3

c.redraw()
.. @+node:ekr.20161117164321.625: *7* << define event callbacks >>
def onClick(p=p):

    g.trace(p)

def onRightClick(p=p):

    g.trace(p)

def onDoubleClick(p=p):

    g.trace(p)
.. @+node:ekr.20161117164321.626: *4* @ignore Scintilla fails
.. @+node:ekr.20161117164321.627: *5* @test expand/contract-pane
import leo.core.leoFrame as leoFrame

# Do nothing when run externally.
if g.app.isExternalUnitTest:
    self.skipTest('Can not be run externally')
else:
    assert not isinstance(c.frame,leoFrame.NullFrame)
    def closeEnough(f1,f2):
        return abs(f1-f2) < 0.01
    f = c.frame
    ratio,ratio2 = f.ratio,f.secondary_ratio
    table = (
        c.bodyWantsFocusNow,
        c.logWantsFocusNow,
        c.treeWantsFocusNow,
    )
    for func in table:
        name = func.__name__
        func()
        f.contractPane()
        if func == c.logWantsFocusNow:
            assert ratio2 != f.secondary_ratio,'fail 1'
        else:
            assert ratio != f.ratio,'fail 2: %s, %s: %s' % (ratio,f.ratio,name)
        func()
        f.expandPane()
        assert closeEnough(ratio,f.ratio),'fail 3 %s != %s: %s' % (
            ratio,f.ratio,name)
        assert closeEnough(ratio2,f.secondary_ratio),'fail 4 %s != %s: %s' % (
            ratio2,f.secondary_ratio,name)
.. @+node:ekr.20161117164321.628: *5* @test add/delete html comments
w = c.frame.body.wrapper
p = g.findNodeInTree(c,p,'html')
assert p,'no test node'
s = p.b
indent = c.config.getBool('indent_added_comments',default=True)
try:
    i = p.b.find('text')
    assert i > -1,'fail1: %s' % (repr(p.b))
    c.selectPosition(p)
    w.setSelectionRange(i,i+4)
    c.addComments()
    if indent:
        i = p.b.find('<!-- text')
    else:
        i = p.b.find('<!--     text')
    assert i > -1,'fail2: %s' % (repr(p.b))
    c.deleteComments()
    assert p.b == s,'fail3: s\n%s\nresult\n%s' % (repr(s),repr(p.b))
    # Add a comment delim without a blank.
    c.addComments()
    p.b = p.b.replace('<!-- ','<!--')
    i = p.b.find('<!--')
    w.setSelectionRange(i,i+4)
    c.deleteComments()
    assert p.b == s,'fail5: s\n%s\nresult\n%s' % (repr(s),repr(p.b))
finally:
    # print('\n'.join([repr(z) for z in g.splitLines(p.b)]))
    p.b = s
.. @+node:ekr.20161117164321.629: *6* html
@language html
<html>
    text 
</html>
.. @+node:ekr.20161117164321.630: *5* @test add/delete python comments
# Can't be run externally.
w = c.frame.body.wrapper
p = g.findNodeInTree(c,p,'python')
assert p,'no test node'
s = p.b
indent = c.config.getBool('indent_added_comments',default=True)

try:
    i = p.b.find('pass')
    assert i > -1,'fail1: %s' % (repr(p.b))
    c.selectPosition(p)
    w.setSelectionRange(i,i+4)
    c.addComments()
    if indent:
        i = p.b.find('# pass')
    else:
        i = p.b.find('#     pass')
    assert i > -1,'fail2: %s' % (repr(p.b))
    c.deleteComments()
    assert p.b == s,'fail3: %s' % (repr(p.b))
    # Add a comment delim without a blank.
    c.addComments()
    p.b = p.b.replace('# pass','#pass')
    i = p.b.find('#')
    w.setSelectionRange(i,i+4)
    c.deleteComments()
    assert p.b == s,'fail5: s\n%s\nresult\n%s' % (repr(s),repr(p.b))
finally:
    # print('\n'.join([repr(z) for z in g.splitLines(p.b)]))
    p.b = s
.. @+node:ekr.20161117164321.631: *6* python
@language python

def spam():
    pass

# after
.. @+node:ekr.20161117164321.632: *5* @test backward-kill-paragraph
c.testManager.runEditCommandTest(p)
.. @+node:ekr.20161117164321.633: *6* work
Americans live in the most severe weather-prone country on Earth. Each year,
Americans cope with an average of 10,000 thunderstorms, 2,500 floods, 1,000
tornadoes, as well as an average of 6 deadly hurricanes. Potentially deadly
weather impacts every American. Communities can now rely on the National Weather
Service’s StormReady program to help them guard against the ravages of Mother
Nature.
to around 500 deaths per year and nearly $14 billion in damage. StormReady, a
program started in 1999 in Tulsa, OK, helps arm America's communities with the
communication and safety skills needed to save lives and property– before and
during the event. StormReady helps community leaders and emergency managers
strengthen local safety programs.

StormReady communities are better prepared to save lives from the onslaught of
severe weather through better planning, education, and awareness. No community
is storm proof, but StormReady can help communities save lives. Does StormReady
make a difference?
.. @+node:ekr.20161117164321.634: *6* before sel=9.0,9.0
Americans live in the most severe weather-prone country on Earth. Each year,
Americans cope with an average of 10,000 thunderstorms, 2,500 floods, 1,000
tornadoes, as well as an average of 6 deadly hurricanes. Potentially deadly
weather impacts every American. Communities can now rely on the National Weather
Service’s StormReady program to help them guard against the ravages of Mother
Nature.

Some 90% of all presidentially declared disasters are weather related, leading
to around 500 deaths per year and nearly $14 billion in damage. StormReady, a
program started in 1999 in Tulsa, OK, helps arm America's communities with the
communication and safety skills needed to save lives and property– before and
during the event. StormReady helps community leaders and emergency managers
strengthen local safety programs.

StormReady communities are better prepared to save lives from the onslaught of
severe weather through better planning, education, and awareness. No community
is storm proof, but StormReady can help communities save lives. Does StormReady
make a difference?
.. @+node:ekr.20161117164321.635: *6* after sel=7.0,7.0
Americans live in the most severe weather-prone country on Earth. Each year,
Americans cope with an average of 10,000 thunderstorms, 2,500 floods, 1,000
tornadoes, as well as an average of 6 deadly hurricanes. Potentially deadly
weather impacts every American. Communities can now rely on the National Weather
Service’s StormReady program to help them guard against the ravages of Mother
Nature.
to around 500 deaths per year and nearly $14 billion in damage. StormReady, a
program started in 1999 in Tulsa, OK, helps arm America's communities with the
communication and safety skills needed to save lives and property– before and
during the event. StormReady helps community leaders and emergency managers
strengthen local safety programs.

StormReady communities are better prepared to save lives from the onslaught of
severe weather through better planning, education, and awareness. No community
is storm proof, but StormReady can help communities save lives. Does StormReady
make a difference?
.. @+node:ekr.20161117164321.636: *5* @test center-region
@pagewidth 70 # Required for unit test.

c.testManager.runEditCommandTest(p)
.. @+node:ekr.20161117164321.637: *6* work
Some 90% of all presidentially declared disasters are weather related,
leading to around 500 deaths per year and nearly $14 billion in damage.
         StormReady, a program started in 1999 in Tulsa, OK,
  helps arm America's communities with the communication and safety
skills needed to save lives and property– before and during the event.
StormReady helps community leaders and emergency managers strengthen local safety programs.
.. @+node:ekr.20161117164321.638: *6* before sel=1.0,7.0
Some 90% of all presidentially declared disasters are weather related,
leading to around 500 deaths per year and nearly $14 billion in damage.
StormReady, a program started in 1999 in Tulsa, OK,
helps arm America's communities with the communication and safety
skills needed to save lives and property– before and during the event.
StormReady helps community leaders and emergency managers strengthen local safety programs.
.. @+node:ekr.20161117164321.639: *6* after sel=1.0,7.0
Some 90% of all presidentially declared disasters are weather related,
leading to around 500 deaths per year and nearly $14 billion in damage.
         StormReady, a program started in 1999 in Tulsa, OK,
  helps arm America's communities with the communication and safety
skills needed to save lives and property– before and during the event.
StormReady helps community leaders and emergency managers strengthen local safety programs.
.. @+node:ekr.20161117164321.640: *5* @test downcase-region
c.testManager.runEditCommandTest(p)
assert g.app.unitTestDict.get('colorized')
.. @+node:ekr.20161117164321.641: *6* work
Americans live in the most severe weather-prone country on Earth. Each year, Americans cope with an average of 10,000 thunderstorms, 2,500 floods, 1,000 tornadoes, as well as an average of 6 deadly hurricanes. Potentially deadly weather impacts every American. Communities can now rely on the National Weather Service’s StormReady program to help them guard against the ravages of Mother Nature.

some 90% of all presidentially declared disasters are weather related, leading to around 500 deaths per year and nearly $14 billion in damage. stormready, a program started in 1999 in tulsa, ok, helps arm america's communities with the communication and safety skills needed to save lives and property– before and during the event. stormready helps community leaders and emergency managers strengthen local safety programs.

StormReady communities are better prepared to save lives from the onslaught of severe weather through better planning, education, and awareness. No community is storm proof, but StormReady can help communities save lives. Does StormReady make a difference?
.. @+node:ekr.20161117164321.642: *6* before sel=3.0,4.0
Americans live in the most severe weather-prone country on Earth. Each year, Americans cope with an average of 10,000 thunderstorms, 2,500 floods, 1,000 tornadoes, as well as an average of 6 deadly hurricanes. Potentially deadly weather impacts every American. Communities can now rely on the National Weather Service’s StormReady program to help them guard against the ravages of Mother Nature.

Some 90% of all presidentially declared disasters are weather related, leading to around 500 deaths per year and nearly $14 billion in damage. StormReady, a program started in 1999 in Tulsa, OK, helps arm America's communities with the communication and safety skills needed to save lives and property– before and during the event. StormReady helps community leaders and emergency managers strengthen local safety programs.

StormReady communities are better prepared to save lives from the onslaught of severe weather through better planning, education, and awareness. No community is storm proof, but StormReady can help communities save lives. Does StormReady make a difference?
.. @+node:ekr.20161117164321.643: *6* after sel=3.0,4.0
Americans live in the most severe weather-prone country on Earth. Each year, Americans cope with an average of 10,000 thunderstorms, 2,500 floods, 1,000 tornadoes, as well as an average of 6 deadly hurricanes. Potentially deadly weather impacts every American. Communities can now rely on the National Weather Service’s StormReady program to help them guard against the ravages of Mother Nature.

some 90% of all presidentially declared disasters are weather related, leading to around 500 deaths per year and nearly $14 billion in damage. stormready, a program started in 1999 in tulsa, ok, helps arm america's communities with the communication and safety skills needed to save lives and property– before and during the event. stormready helps community leaders and emergency managers strengthen local safety programs.

StormReady communities are better prepared to save lives from the onslaught of severe weather through better planning, education, and awareness. No community is storm proof, but StormReady can help communities save lives. Does StormReady make a difference?
.. @+node:ekr.20161117164321.644: *5* @test kill-paragraph
c.testManager.runEditCommandTest(p)
.. @+node:ekr.20161117164321.645: *6* work
Americans live in the most severe weather-prone country on Earth. Each year,
Americans cope with an average of 10,000 thunderstorms, 2,500 floods, 1,000
tornadoes, as well as an average of 6 deadly hurricanes. Potentially deadly
weather impacts every American. Communities can now rely on the National Weather
Service’s StormReady program to help them guard against the ravages of Mother
Nature.



StormReady communities are better prepared to save lives from the onslaught of
severe weather through better planning, education, and awareness. No community
is storm proof, but StormReady can help communities save lives. Does StormReady
make a difference?
.. @+node:ekr.20161117164321.646: *6* before sel=9.0,9.0
Americans live in the most severe weather-prone country on Earth. Each year,
Americans cope with an average of 10,000 thunderstorms, 2,500 floods, 1,000
tornadoes, as well as an average of 6 deadly hurricanes. Potentially deadly
weather impacts every American. Communities can now rely on the National Weather
Service’s StormReady program to help them guard against the ravages of Mother
Nature.

Some 90% of all presidentially declared disasters are weather related, leading
to around 500 deaths per year and nearly $14 billion in damage. StormReady, a
program started in 1999 in Tulsa, OK, helps arm America's communities with the
communication and safety skills needed to save lives and property– before and
during the event. StormReady helps community leaders and emergency managers
strengthen local safety programs.

StormReady communities are better prepared to save lives from the onslaught of
severe weather through better planning, education, and awareness. No community
is storm proof, but StormReady can help communities save lives. Does StormReady
make a difference?
.. @+node:ekr.20161117164321.647: *6* after sel=8.0,8.0
Americans live in the most severe weather-prone country on Earth. Each year,
Americans cope with an average of 10,000 thunderstorms, 2,500 floods, 1,000
tornadoes, as well as an average of 6 deadly hurricanes. Potentially deadly
weather impacts every American. Communities can now rely on the National Weather
Service’s StormReady program to help them guard against the ravages of Mother
Nature.



StormReady communities are better prepared to save lives from the onslaught of
severe weather through better planning, education, and awareness. No community
is storm proof, but StormReady can help communities save lives. Does StormReady
make a difference?
.. @+node:ekr.20161117164321.648: *5* @test selfInsertCommand-1
@first # -*- coding: utf-8 -*-
@language python

try:
    ec = c.editCommands ; w = c.frame.body.wrapper
    s = w.getAllText()

    # This strings tests unicode, paren matching, and auto-indentation.
    u = g.u('(a\u00c9\u03a9B\u3045\u4e7cz):\n') # '(aÉ©BE|cz):\n'
    u = g.u('(pdq):\n')
    w.setInsertPoint(len(s))
    for char in u:
        stroke = g.choose(char=='\n','Return',char)
        event = g.app.gui.create_key_event(c,char,stroke,w)
        ec.selfInsertCommand(event)
    result = w.getAllText()
    #g.trace('result',repr(result))
    assert result.endswith('    '),'result:\n%s' % result
    # Test of autocompleter.
finally:
    if 1:
        w.setAllText(s)
        p.setBodyString(s)
        # g.trace(repr(s))
        c.recolor()

# end:
.. @+node:ekr.20161117164321.649: *5* @test upcase-region
c.testManager.runEditCommandTest(p)
assert g.app.unitTestDict.get('colorized')
.. @+node:ekr.20161117164321.650: *6* work
Americans live in the most severe weather-prone country on Earth. Each year, Americans cope with an average of 10,000 thunderstorms, 2,500 floods, 1,000 tornadoes, as well as an average of 6 deadly hurricanes. Potentially deadly weather impacts every American. Communities can now rely on the National Weather Service’s StormReady program to help them guard against the ravages of Mother Nature.

SOME 90% OF ALL PRESIDENTIALLY DECLARED DISASTERS ARE WEATHER RELATED, LEADING TO AROUND 500 DEATHS PER YEAR AND NEARLY $14 BILLION IN DAMAGE. STORMREADY, A PROGRAM STARTED IN 1999 IN TULSA, OK, HELPS ARM AMERICA'S COMMUNITIES WITH THE COMMUNICATION AND SAFETY SKILLS NEEDED TO SAVE LIVES AND PROPERTY– BEFORE AND DURING THE EVENT. STORMREADY HELPS COMMUNITY LEADERS AND EMERGENCY MANAGERS STRENGTHEN LOCAL SAFETY PROGRAMS.

StormReady communities are better prepared to save lives from the onslaught of severe weather through better planning, education, and awareness. No community is storm proof, but StormReady can help communities save lives. Does StormReady make a difference?
.. @+node:ekr.20161117164321.651: *6* before sel=3.0,4.0
Americans live in the most severe weather-prone country on Earth. Each year, Americans cope with an average of 10,000 thunderstorms, 2,500 floods, 1,000 tornadoes, as well as an average of 6 deadly hurricanes. Potentially deadly weather impacts every American. Communities can now rely on the National Weather Service’s StormReady program to help them guard against the ravages of Mother Nature.

Some 90% of all presidentially declared disasters are weather related, leading to around 500 deaths per year and nearly $14 billion in damage. StormReady, a program started in 1999 in Tulsa, OK, helps arm America's communities with the communication and safety skills needed to save lives and property– before and during the event. StormReady helps community leaders and emergency managers strengthen local safety programs.

StormReady communities are better prepared to save lives from the onslaught of severe weather through better planning, education, and awareness. No community is storm proof, but StormReady can help communities save lives. Does StormReady make a difference?
.. @+node:ekr.20161117164321.652: *6* after sel=3.0,4.0
Americans live in the most severe weather-prone country on Earth. Each year, Americans cope with an average of 10,000 thunderstorms, 2,500 floods, 1,000 tornadoes, as well as an average of 6 deadly hurricanes. Potentially deadly weather impacts every American. Communities can now rely on the National Weather Service’s StormReady program to help them guard against the ravages of Mother Nature.

SOME 90% OF ALL PRESIDENTIALLY DECLARED DISASTERS ARE WEATHER RELATED, LEADING TO AROUND 500 DEATHS PER YEAR AND NEARLY $14 BILLION IN DAMAGE. STORMREADY, A PROGRAM STARTED IN 1999 IN TULSA, OK, HELPS ARM AMERICA'S COMMUNITIES WITH THE COMMUNICATION AND SAFETY SKILLS NEEDED TO SAVE LIVES AND PROPERTY– BEFORE AND DURING THE EVENT. STORMREADY HELPS COMMUNITY LEADERS AND EMERGENCY MANAGERS STRENGTHEN LOCAL SAFETY PROGRAMS.

StormReady communities are better prepared to save lives from the onslaught of severe weather through better planning, education, and awareness. No community is storm proof, but StormReady can help communities save lives. Does StormReady make a difference?
.. @+node:ekr.20161117164321.653: *5* @test reformat-paragraph paragraph 1 of 3
# Required when running tests externally
@language plain
@pagewidth 40
@tabwidth 8

c.testManager.runEditCommandTest(p)
.. @+node:ekr.20161117164321.654: *6* work
Americans live in the most severe
weather-prone country on Earth. Each
year, Americans cope with an average of
10,000 thunderstorms, 2,500 floods,
1,000 tornadoes, as well as an average
of 6 deadly hurricanes. Potentially
deadly weather impacts every American.
Communities can now rely on the National
Weather Service’s StormReady program to
help them guard against the ravages of
Mother Nature.

Some 90% of all presidentially declared disasters are weather related, leading to around 500 deaths per year and nearly $14 billion in damage. StormReady, a program started in 1999 in Tulsa, OK, helps arm America's communities with the communication and safety skills needed to save lives and property– before and during the event. StormReady helps community leaders and emergency managers strengthen local safety programs.

StormReady communities are better prepared to save lives from the onslaught of severe weather through better planning, education, and awareness. No community is storm proof, but StormReady can help communities save lives. Does StormReady make a difference?

Last paragraph.
.. @+node:ekr.20161117164321.655: *6* before sel=1.0,1.0
Americans live in the most severe weather-prone country on Earth. Each year, Americans cope with an average of 10,000 thunderstorms, 2,500 floods, 1,000 tornadoes, as well as an average of 6 deadly hurricanes. Potentially deadly weather impacts every American. Communities can now rely on the National Weather Service’s StormReady program to help them guard against the ravages of Mother Nature.

Some 90% of all presidentially declared disasters are weather related, leading to around 500 deaths per year and nearly $14 billion in damage. StormReady, a program started in 1999 in Tulsa, OK, helps arm America's communities with the communication and safety skills needed to save lives and property– before and during the event. StormReady helps community leaders and emergency managers strengthen local safety programs.

StormReady communities are better prepared to save lives from the onslaught of severe weather through better planning, education, and awareness. No community is storm proof, but StormReady can help communities save lives. Does StormReady make a difference?

Last paragraph.
.. @+node:ekr.20161117164321.656: *6* after sel=11.14,11.14
Americans live in the most severe
weather-prone country on Earth. Each
year, Americans cope with an average of
10,000 thunderstorms, 2,500 floods,
1,000 tornadoes, as well as an average
of 6 deadly hurricanes. Potentially
deadly weather impacts every American.
Communities can now rely on the National
Weather Service’s StormReady program to
help them guard against the ravages of
Mother Nature.

Some 90% of all presidentially declared disasters are weather related, leading to around 500 deaths per year and nearly $14 billion in damage. StormReady, a program started in 1999 in Tulsa, OK, helps arm America's communities with the communication and safety skills needed to save lives and property– before and during the event. StormReady helps community leaders and emergency managers strengthen local safety programs.

StormReady communities are better prepared to save lives from the onslaught of severe weather through better planning, education, and awareness. No community is storm proof, but StormReady can help communities save lives. Does StormReady make a difference?

Last paragraph.
.. @+node:ekr.20161117164321.657: *5* @test reformat-paragraph paragraph 2 of 3
# Required when running tests externally
@language plain
@pagewidth 40
@tabwidth 8

c.testManager.runEditCommandTest(p)
.. @+node:ekr.20161117164321.658: *6* work
Americans live in the most severe
weather-prone country on Earth. Each
year, Americans cope with an average of
10,000 thunderstorms, 2,500 floods,
1,000 tornadoes, as well as an average
of 6 deadly hurricanes. Potentially
deadly weather impacts every American.
Communities can now rely on the National
Weather Service’s StormReady program to
help them guard against the ravages of
Mother Nature.

Some 90% of all presidentially declared
disasters are weather related, leading
to around 500 deaths per year and nearly
$14 billion in damage. StormReady, a
program started in 1999 in Tulsa, OK,
helps arm America's communities with the
communication and safety skills needed
to save lives and property– before and
during the event. StormReady helps
community leaders and emergency managers
strengthen local safety programs.

StormReady communities are better prepared to save lives from the onslaught of severe weather through better planning, education, and awareness. No community is storm proof, but StormReady can help communities save lives. Does StormReady make a difference?

Last paragraph.
.. @+node:ekr.20161117164321.659: *6* before sel=13.0,13.0
Americans live in the most severe
weather-prone country on Earth. Each
year, Americans cope with an average of
10,000 thunderstorms, 2,500 floods,
1,000 tornadoes, as well as an average
of 6 deadly hurricanes. Potentially
deadly weather impacts every American.
Communities can now rely on the National
Weather Service’s StormReady program to
help them guard against the ravages of
Mother Nature.

Some 90% of all presidentially declared disasters are weather related, leading to around 500 deaths per year and nearly $14 billion in damage. StormReady, a program started in 1999 in Tulsa, OK, helps arm America's communities with the communication and safety skills needed to save lives and property– before and during the event. StormReady helps community leaders and emergency managers strengthen local safety programs.

StormReady communities are better prepared to save lives from the onslaught of severe weather through better planning, education, and awareness. No community is storm proof, but StormReady can help communities save lives. Does StormReady make a difference?

Last paragraph.
.. @+node:ekr.20161117164321.660: *6* after sel=23.33,23.33
Americans live in the most severe
weather-prone country on Earth. Each
year, Americans cope with an average of
10,000 thunderstorms, 2,500 floods,
1,000 tornadoes, as well as an average
of 6 deadly hurricanes. Potentially
deadly weather impacts every American.
Communities can now rely on the National
Weather Service’s StormReady program to
help them guard against the ravages of
Mother Nature.

Some 90% of all presidentially declared
disasters are weather related, leading
to around 500 deaths per year and nearly
$14 billion in damage. StormReady, a
program started in 1999 in Tulsa, OK,
helps arm America's communities with the
communication and safety skills needed
to save lives and property– before and
during the event. StormReady helps
community leaders and emergency managers
strengthen local safety programs.

StormReady communities are better prepared to save lives from the onslaught of severe weather through better planning, education, and awareness. No community is storm proof, but StormReady can help communities save lives. Does StormReady make a difference?

Last paragraph.
.. @+node:ekr.20161117164321.661: *5* @test reformat-paragraph paragraph 3 of 3
# Required when running tests externally
@language plain
@pagewidth 40
@tabwidth 8

c.testManager.runEditCommandTest(p)
.. @+node:ekr.20161117164321.662: *6* work
Americans live in the most severe
weather-prone country on Earth. Each
year, Americans cope with an average of
10,000 thunderstorms, 2,500 floods,
1,000 tornadoes, as well as an average
of 6 deadly hurricanes. Potentially
deadly weather impacts every American.
Communities can now rely on the National
Weather Service’s StormReady program to
help them guard against the ravages of
Mother Nature.

Some 90% of all presidentially declared
disasters are weather related, leading
to around 500 deaths per year and nearly
$14 billion in damage. StormReady, a
program started in 1999 in Tulsa, OK,
helps arm America's communities with the
communication and safety skills needed
to save lives and property– before and
during the event. StormReady helps
community leaders and emergency managers
strengthen local safety programs.

StormReady communities are better
prepared to save lives from the
onslaught of severe weather through
better planning, education, and
awareness. No community is storm proof,
but StormReady can help communities save
lives. Does StormReady make a
difference?

Last paragraph.
.. @+node:ekr.20161117164321.663: *6* before sel=25.10,25.10
Americans live in the most severe
weather-prone country on Earth. Each
year, Americans cope with an average of
10,000 thunderstorms, 2,500 floods,
1,000 tornadoes, as well as an average
of 6 deadly hurricanes. Potentially
deadly weather impacts every American.
Communities can now rely on the National
Weather Service’s StormReady program to
help them guard against the ravages of
Mother Nature.

Some 90% of all presidentially declared
disasters are weather related, leading
to around 500 deaths per year and nearly
$14 billion in damage. StormReady, a
program started in 1999 in Tulsa, OK,
helps arm America's communities with the
communication and safety skills needed
to save lives and property– before and
during the event. StormReady helps
community leaders and emergency managers
strengthen local safety programs.

StormReady communities are better prepared to save lives from the onslaught of severe weather through better planning, education, and awareness. No community is storm proof, but StormReady can help communities save lives. Does StormReady make a difference?

Last paragraph.
.. @+node:ekr.20161117164321.664: *6* after sel=32.11,32.11
Americans live in the most severe
weather-prone country on Earth. Each
year, Americans cope with an average of
10,000 thunderstorms, 2,500 floods,
1,000 tornadoes, as well as an average
of 6 deadly hurricanes. Potentially
deadly weather impacts every American.
Communities can now rely on the National
Weather Service’s StormReady program to
help them guard against the ravages of
Mother Nature.

Some 90% of all presidentially declared
disasters are weather related, leading
to around 500 deaths per year and nearly
$14 billion in damage. StormReady, a
program started in 1999 in Tulsa, OK,
helps arm America's communities with the
communication and safety skills needed
to save lives and property– before and
during the event. StormReady helps
community leaders and emergency managers
strengthen local safety programs.

StormReady communities are better
prepared to save lives from the
onslaught of severe weather through
better planning, education, and
awareness. No community is storm proof,
but StormReady can help communities save
lives. Does StormReady make a
difference?

Last paragraph.
.. @+node:ekr.20161117164321.665: *5* @test most toggle commands
k = c.k
colorizer = c.frame.body.getColorizer()
ed = c.editCommands

# These don't set ivars
    # 'toggle-active-pane'),
    # 'toggle-angle-brackets',
    # 'toggle-input-state'),
    # 'toggle-mini-buffer'),
    # 'toggle-split-direction'),

table = [
    (k,'abbrevOn','toggle-abbrev-mode'),
    (ed,'extendMode','toggle-extend-mode'),
]

# Not valid for external tests.
table2 = [
    (k,'enable_autocompleter','toggle-autocompleter'),
    (k,'enable_calltips','toggle-calltips'),
    (c,'sparse_find','toggle-find-collapses-nodes'),
    (colorizer,'showInvisibles','toggle-invisibles'),
    (c,'sparse_move','toggle-sparse-move'),
]

if not g.app.isExternalUnitTest:
    table.extend(table2)

for obj,ivar,command in table:
    val1 = getattr(obj,ivar)
    try:
        k.simulateCommand(command)
        val2 = getattr(obj,ivar)
        assert val2 == (not val1),'failed 1 %s' % command
        k.simulateCommand(command)
        val3 = getattr(obj,ivar)
        assert val3 == val1,'failed 2 %s' % command
        # print('pass',command)
    finally:
        setattr(obj,ivar,val1)
.. @+node:ekr.20161117164321.666: *5* @suite Edit body tests
# Create unit tests in g.app.scriptDict["suite"]

suite = c.testManager.makeEditBodySuite(p)

# g.app.scriptDict['suite'] = suite
.. @+node:ekr.20161117164321.667: *6* editBodyTests
@language plain
@

The names of child nodes are the names of commander methods to be called to do the test.

Each child node will in turn have two or more children:

- a "before" node
- an "after" node
- an optional selection node containing two lines giving the selection range in Tk coordinates.
- An optional insert node containing one line giving the insert point in Tk coordinates.
.. @+node:ekr.20161117164321.668: *7* tempNode
.. @+node:ekr.20161117164321.669: *7* addComments
.. @+node:ekr.20161117164321.670: *8* before
@language python

def addCommentTest():

    if 1:
        a = 2
        b = 3

    pass
.. @+node:ekr.20161117164321.671: *8* after
@language python

def addCommentTest():

    # if 1:
        # a = 2
        # b = 3

    pass
.. @+node:ekr.20161117164321.672: *8* selection
5.0
7.8
.. @+node:ekr.20161117164321.673: *7* convertAllBlanks
.. @+node:ekr.20161117164321.674: *8* before
@tabwidth -4

line 1
    line 2
      line 3
line4
.. @+node:ekr.20161117164321.675: *8* after
@tabwidth -4

line 1
	line 2
	  line 3
line4
.. @+node:ekr.20161117164321.676: *8* selection
1.0
6.5
.. @+node:ekr.20161117164321.677: *7* convertAllTabs
.. @+node:ekr.20161117164321.678: *8* before
@tabwidth -4

line 1
	line 2
	  line 3
line4
.. @+node:ekr.20161117164321.679: *8* after
@tabwidth -4

line 1
    line 2
      line 3
line4
.. @+node:ekr.20161117164321.680: *8* selection
1.0
6.5
.. @+node:ekr.20161117164321.681: *7* convertBlanks
.. @+node:ekr.20161117164321.682: *8* before
@tabwidth -4

line 1
    line 2
      line 3
line4
.. @+node:ekr.20161117164321.683: *8* after
@tabwidth -4

line 1
	line 2
	  line 3
line4
.. @+node:ekr.20161117164321.684: *8* selection
1.0
6.5
.. @+node:ekr.20161117164321.685: *7* convertTabs
.. @+node:ekr.20161117164321.686: *8* before
@tabwidth -4

line 1
	line 2
	  line 3
line4
.. @+node:ekr.20161117164321.687: *8* after
@tabwidth -4

line 1
    line 2
      line 3
line4
.. @+node:ekr.20161117164321.688: *8* selection
1.0
6.5
.. @+node:ekr.20161117164321.689: *7* dedentBody
.. @+node:ekr.20161117164321.690: *8* before
line 1
    line 2
    line 3
line 4
.. @+node:ekr.20161117164321.691: *8* after
line 1
line 2
line 3
line 4
.. @+node:ekr.20161117164321.692: *8* selection
2.0
3.5
.. @+node:ekr.20161117164321.693: *7* deleteComments
# created by new add-comments
.. @+node:ekr.20161117164321.694: *8* before
@language python

def deleteCommentTest():

#     if 1:
#         a = 2
#         b = 3

    pass
.. @+node:ekr.20161117164321.695: *8* after
@language python

def deleteCommentTest():

    if 1:
        a = 2
        b = 3

    pass
.. @+node:ekr.20161117164321.696: *8* selection
5.0
7.8
.. @+node:ekr.20161117164321.697: *7* deleteComments
# created by old and new add-comments.
.. @+node:ekr.20161117164321.698: *8* before
@language python

def deleteCommentTest():

#     if 1:
#         a = 2
#         b = 3

    # if 1:
        # a = 2
        # b = 3

    pass
.. @+node:ekr.20161117164321.699: *8* after
@language python

def deleteCommentTest():

    if 1:
        a = 2
        b = 3

    if 1:
        a = 2
        b = 3

    pass
.. @+node:ekr.20161117164321.700: *8* selection
5.0
12.8
.. @+node:ekr.20161117164321.701: *7* extract test1
.. @+node:ekr.20161117164321.702: *8* before
before
    << section >>
    sec line 1
        sec line 2 indented
sec line 3
after
.. @+node:ekr.20161117164321.703: *8* after
before
    << section >>
after
.. @+node:ekr.20161117164321.704: *9* << section >> @nonl
sec line 1
    sec line 2 indented
sec line 3
.. @+node:ekr.20161117164321.705: *8* selection
2.0
5.10
.. @+node:ekr.20161117164321.706: *7* extract test2
.. @+node:ekr.20161117164321.707: *8* before
before
    << section >>
    sec line 1
        sec line 2 indented
sec line 3
after
.. @+node:ekr.20161117164321.708: *8* after
before
    << section >>
    sec line 1
        sec line 2 indented
sec line 3
after
.. @+node:ekr.20161117164321.709: *8* selection
2.0
2.16
.. @+node:ekr.20161117164321.710: *7* extractSection test1
.. @+node:ekr.20161117164321.711: *8* before
before
    << section >>
    sec line 1
        sec line 2 indented
sec line 3
after
.. @+node:ekr.20161117164321.712: *8* after
before
    << section >>
after
.. @+node:ekr.20161117164321.713: *9* << section >> @nonl
sec line 1
    sec line 2 indented
sec line 3
.. @+node:ekr.20161117164321.714: *8* selection
2.0
5.10
.. @+node:ekr.20161117164321.715: *7* extractSection test2
.. @+node:ekr.20161117164321.716: *8* before
before
    << section >>
    sec line 1
        sec line 2 indented
sec line 3
after
.. @+node:ekr.20161117164321.717: *8* after
before
    << section >>
    sec line 1
        sec line 2 indented
sec line 3
after
.. @+node:ekr.20161117164321.718: *8* selection
2.0
2.16
.. @+node:ekr.20161117164321.719: *4* @ignore Unit tests for settings
.. @+node:ekr.20161117164321.720: *5* print dicts unit tests
.. @+node:ekr.20161117164321.721: *6* @test printMenusList
def printMenusList(aList,level=0):
    
    for z in aList:
        a,b,c = z
        print('*** kind',a)
        if type(b) in (type(()),type([])):
            for z2 in b:
                a1,b1,c1 = z2
                if a1.startswith('@menu') and type(b1) in (type(()),type([])):
                    print()
                    print('*** inner menu: %s' % (level+1))
                    print(a1)
                    for z3 in b1:
                        print(z3)
                    if c1: print(c1)
                else:
                    print(z2)
            if c: print(c)
        else:
            print(b)
        print()
        break #
        
printMenusList(c.config.getMenusList())
       
.. @+node:ekr.20161117164321.722: *6* @test printInverseBindingDict
print('\ninverseBindingDict...\n')

d = c.k.computeInverseBindingDict()

for key in sorted(list(d.keys())):

    if 1 == len(d.get(key)):
        print(key,d.get(key))
    else:
        print()
        print(key)
        print(d.get(key))
        print()
.. @+node:ekr.20161117164321.723: *6* @test printBindingsDict
import leo.core.leoConfig as leoConfig # for ShortcutInfo
    
partial = True

d = c.k.bindingsDict
    # Keys are shortcuts; values are *lists* leoConfig.ShortcutInfo objects.
    
print('\nk.bindingsDict%s...\n' % ' (partial)' if partial else '')
    
for key in list(sorted(d.keys())):
    aList = d.get(key,[])
    for b in aList:
        assert isinstance(b,leoConfig.ShortcutInfo)
        if not partial or b.kind != 'leosettings.leo':
            print(b)
.. @+node:ekr.20161117164321.724: *6* @test printMasterBindingsDict
partial = True

panes = ('all','body','button','log','tree','text',
    'command','insert','overwrite',)

d = c.k.masterBindingsDict
    # Keys are scope names (in panes) or mode names.
    # Values are dicts:
        # keys are strokes; values are leoConfig.ShortcutInfo objects.
        
print('\nk.masterBindingsDict%s...\n' % ' (partial)' if partial else '')

for pane in sorted(list(d.keys())):
    kind = 'pane' if pane in panes else 'mode'
    print('%s: %s...' % (kind,pane))
    d2 = d.get(pane)
    for stroke in sorted(list(d2.keys())):
        b = d2.get(stroke)
        if not partial or b.kind != 'leosettings.leo':
            print('%6s %25s %17s %s' % (b.pane,stroke,b.kind,b.commandName))
            assert b.pane == pane
            assert b.stroke == stroke
    print()
.. @+node:ekr.20161117164321.725: *5* passed
.. @+node:ekr.20161117164321.726: *6* @test mode-related info
@

g.app.config.modeCommandsDict
    Keys are command names: enter-x-mode.
    Values are inner dictionaries:
        Keys are command names, values are lists of ShortcutInfo nodes.
@c

d = g.app.config.modeCommandsDict
    
for key in sorted(d.keys()):
    print('*** mode ***',key)
    d2 = d.get(key)
    for key2 in sorted(d2.keys()):
        aList = d2.get(key2)
        print(key2)
        for si in aList:
            print('   ',si)
.. @+node:ekr.20161117164321.727: *6* @test types of contents of settings dicts
@
ivar                    Keys                Values
----                    ----                ------
c.commandsDict          command names (1)   functions
k.inverseCommandsDict   func.__name__       command names
k.bindingsDict          shortcuts           list of ShortcutInfo objects
k.masterBindingsDict    scope names (2)     inner masterBindingDicts (3)
k.masterGuiBindingsDict strokes             list of widgets in which stoke is bound
k.settingsNameDict (4)  settings.lower()    "Real" Tk specifiers
inverseBindingDict (5)  command names       lists of tuples (pane,key)
modeCommandsDict (6)    command name (7)    inner modeCommandsDicts (8)

Notes:
(1) Command names are minibuffer names (strings)
(2) Scope names are 'all','text',etc.
(3) inner masterBindingDicts: Keys are strokes; values are ShortcutInfo objects.
(4) k.settingsNameDict has no inverse.
(5) inverseBindingDict is **not** an ivar: it is computed by k.computeInverseBindingDict.
(6) A global dict: g.app.gui.modeCommandsDict
(7) enter-x-command
(8) Keys are command names, values are lists of ShortcutInfo objects.
@c

si_type = c.k.ShortcutInfo
disabled_func_type = None # Should be any bound method.
k = c.k

@others

test_dict_of_objects(c.commandsDict,type('s'),disabled_func_type,'commandsDict')
test_dict_of_objects(k.inverseCommandsDict,type('s'),type('s'),'inverseCommandsDict')
test_dict_of_lists(k.bindingsDict,si_type,'bindingsDict')
test_dict_of_dicts(k.masterBindingsDict,si_type,'masterBindingsDict')
test_dict_of_lists(k.masterGuiBindingsDict,None,'masterGuiBindingsDict')
test_dict_of_objects(k.settingsNameDict,type('s'),type('s'),'settingsNameDict')
test_dict_of_lists(k.computeInverseBindingDict(),type(tuple()),'inverseBindingDict')

# Test individual dicts separately.
d = g.app.config.modeCommandsDict
test_dict_of_dicts(d,None,'modeCommandsDict')
for key in sorted(d.keys()):
    d2 = d.get(key)
    test_dict_of_lists(d2,si_type,'inner modeCommandsDict')
        # This requires a hack to special-case the
        # '*entry-commands*' and '*command-prompt*' keys.
.. @+node:ekr.20161117164321.728: *7* test_dict_of_dicts
def test_dict_of_dicts(d,theType,tag):

    assert d,tag

    for key in d.keys():
        d2 = d.get(key)
        assert type(d2) == type({})
        for key in d2.keys():
            obj = d2.get(key)
            if theType:
                assert type(obj) == theType,repr(obj)
.. @+node:ekr.20161117164321.729: *7* test_dict_of_lists
def test_dict_of_lists(d,theType,tag):

    assert d,tag

    for key in d.keys():
        obj = d.get(key)
        if key in ('*entry-commands*','*command-prompt*'):
            # Special case for g.app.config.modeCommandsDict
            assert type(obj)==type([]),repr(obj)
        else:
            assert type(obj) == type([])
            # Don't check types of list elements if theType is None.
            if theType:
                for z in obj:
                    assert type(z)==theType,'key: %s obj: %s' % (key,repr(obj))
.. @+node:ekr.20161117164321.730: *7* test_dict_of_objects
def test_dict_of_objects(d,keyType,valueType,tag):

    assert d,tag

    for key in d.keys():
        assert type(key) == keyType,repr(key)
        obj = d.get(key)
        # Don't check type of obj if valueType is None.
        if valueType:
            assert type(obj) == valueType,'\nobj: %s\nvalueType: %s' % (repr(obj),valueType)
.. @+node:ekr.20161117164321.731: *7* Unused
# import types
# types.ListType does not exist in Python 3.x.
# assert isinstance(aList,list().__class__)
.. @+node:ekr.20161117164321.732: *6* @test merge_settings_dicts
@others

# import os ; os.system('cls')
    
d1 = g.app.config.immutable_leo_settings_shortcuts_dict
d2 = g.app.config.immutable_my_leo_settings_shortcuts_dict
d3 = g.app.config.merge_settings_dicts(d1,d2)

if False:
    patterns = (
        'backward-find-character-extend-selection',
    )
    for pattern in patterns:
        print(dump_dict(d1,pattern,tag='d1'))
        print(dump_dict(d2,pattern,tag='d2'))
        print(dump_dict(d3,pattern,tag='d3'))

test(d1,d2,d3)
.. @+node:ekr.20161117164321.733: *7* dump & dump_dict (@test merge_settings_dicts)
def dump(aList,pattern=None,tag=None):
    
    return '\n'.join([repr(z) for z in aList])
    

def dump_dict(d,pattern=None,tag=None):
    
    result = [] # '\ndump of %s...' % (tag)
    
    for key in d.keys():
        if pattern in (key,None):
            result.append(key)
            aList = d.get(key)
            for z in aList:
                result.append('    %s' % (z))
                
    return '\n'.join(result)
.. @+node:ekr.20161117164321.734: *7* test (@test merge_settings_dicts)
def test(old_d,new_d,result_d):
    
    '''Test that result_d is the result of upating old_d with new_d.
    
    This test is tricky: only inverted dicts have ShortcutInfo nodes as keys.'''
    
    invert,uninvert = g.app.config.invert,g.app.config.uninvert

    # Compute the inversions of all the dicts.
    inv_old,inv_new,inv_res = invert(old_d),invert(new_d),invert(result_d)
    
    # Part 1: Ensure we test all keys.
    keys = list(inv_old.keys())
    keys.extend(list(inv_new.keys()))
    keys.extend(list(inv_res.keys()))
    keys = sorted(list(set(keys)))
    assert None not in keys
    for key in inv_old.keys(): assert key in keys,key
    for key in inv_new.keys(): assert key in keys,key
    for key in inv_res.keys(): assert key in keys,key
    
    # Part 2: Carefully test the inverted result.
    def si_name_key(si): return si.commandName or ''

    for key in keys:
        # Compute the *sorted* list of 
        res_list = sorted(inv_res.get(key,[]),key=si_name_key)
        old_list = sorted(inv_old.get(key,[]),key=si_name_key)
        new_list = sorted(inv_new.get(key,[]),key=si_name_key)
        assert res_list,'no res_list.get(%s)' % (key)
        # if new_list: print(key,dump(new_list))
        if new_list:
            assert new_list == res_list,'key %s\nnew:\n%s\nres:\n%s' % (
                key,dump(new_list),dump(res_list))
        else:
            assert old_list == res_list,'key %s\nold:\n%s\nres:\n%s' % (
                key,dump(old_list),dump(res_list))
    
    # Part 3: Test that result_d == uninvert(invert(result_d)).
    # A.  They must have the same keys.
    unv_res = uninvert(inv_res)
    assert sorted(list(result_d.keys())) == sorted(list(unv_res.keys()))

    # B. The values of for each key must match after being sorted.
    def si_stroke_key(si): return si.stroke or ''
        
    for key in sorted(result_d.keys()):
        res_list = sorted(result_d.get(key,[]),key=si_stroke_key)
        unv_list = sorted( unv_res.get(key,[]),key=si_stroke_key)
        assert res_list == unv_list,'key %s\nres:\n%s\nunv:\n%s' % (
            key,dump(res_list),dump(unv_list))
   
.. @+node:ekr.20161117164321.735: *6* @test KeyStroke
ks = c.k.KeyStroke

@others

a1 = ks('a')
a2 = ks('a')
b1 = ks('b')
assert a1 == a2
d = {}
d[a1] = a1.s
d[a2] = a2.s
d[b1] = b1.s

for key in sorted(d):
    print(key,d.get(key))
.. @+node:ekr.20161117164321.736: *6* @test g.TypedDict
d = g.TypedDictOfLists('ks',type('s'),type(9))
d.add('a',1)
d.add('a',2)
d.add('b',3)

print(d)
for s in sorted(d.keys()):
    print(s,d.get(s,[]))

print('after replace...')
d.replace('a',[8,9,10])

for s in sorted(d.keys()):
    print(s,d.get(s,[]))
.. @+node:ekr.20161117164321.737: *4* @ignore unused tests
.. @+node:ekr.20161117164321.738: *5* @@test delete-body-editor 3 times
if 0:
    # This works, but the label sticks, presumably because of a race condition.
    # Therefore, this is best left as a mini-test.

    w = c.frame.body
    for i in range(4):
        print(i)
        w.addEditor()
        w.deleteEditor()
.. @+node:ekr.20161117164321.739: *5* @@test loading .leo file with @file nodes
# test/at-file-test.leo contains tnodeList.
# test/at-file-test.py contains file-like sentinels.
g.app.unitTestDict={}
fn = g.os_path_finalize_join(g.app.loadDir,'..','test','unittest','at-file-test.leo')

try:
    c2 = g.openWithFileName(fn,c)
    assert c2
    assert c2.changed,'not changed'
    p2 = g.findNodeAnywhere(c2,'@file at-file-test.py')
    assert p2,'no p2'
    assert p2.isDirty(),'not dirty'
    assert g.app.unitTestDict.get('read-convert'),'not converted'
    ok = True
finally:
    if True: # and ok:
        # Close the frame without prompt.
        g.app.destroyWindow(c2.frame)
        c.setLog()
        c.bodyWantsFocus()
.. @+node:ekr.20161117164321.740: *5* @@test tkBody.onClick
w = c.frame.body.bodyCtrl
y = 10
for x in range(0,100,10):
    event = g.app.gui.create_key_event(c,None,None,w,x=x,y=y)
    c.frame.body.onClick(event)
.. @+node:ekr.20161117164321.741: *5* @@test typing in headline recomputes width
# getWidth no longer exists.
k = c.keyHandler
h = 'Test headline abc'
p = c.testManager.findNodeAnywhere(h)
assert p,'node not found: %s' % h
c.redraw(p) # To make node visible
c.frame.tree.editLabel(p)
w = c.edit_widget(p)
try:
    assert w
    g.app.gui.set_focus(c,w)
    w2 = g.app.gui.get_focus(c)
    # assert w == w2 or hasattr(w,'widget') and w.widget == w2,'w: %s\nw2: %s' % (w,w2)
    w.setSelectionRange('end','end')
    n = w.getWidth()
    g.app.gui.event_generate(c,'X','Shift+X',w)
    g.app.gui.event_generate(c,'Y','Shift+Y',w)
    g.app.gui.event_generate(c,'Z','Shift+Z',w)
    g.app.gui.event_generate(c,'\n','Return',w)
    w.update()
    assert w, 'fail 2'
   
finally:
    if 1:
        c.setHeadString(p,h) # Essential
        c.redraw(p)
.. @+node:ekr.20161117164321.742: *5* @@test write .leo file with @ignore node
assert p.firstChild(), 'no child node'
assert p.firstChild().b.startswith('@ignore'), 'No @ignore in child'
ok = c.fileCommands.write_Leo_file(
    'file-name',outlineOnlyFlag=True,toString=True,toOPML=False)
assert ok, 'error writing file'
count = 0
s = g.app.write_Leo_file_string
for line in g.splitLines(s):
    if line.find('@ignore') != -1:
        count += 1
assert count >=1, "not enough @ignore's in written file: count: %s, lines:\n%s" % (count,s)
.. @+node:ekr.20161117164321.743: *6* child
@ignore # Test that this node gets written.
.. @+node:ekr.20161117164321.744: *5* general file stuff
.. @+node:ekr.20161117164321.745: *6* @@test raw file copy
import tempfile
import os

s = 'Select the following string: वादक. Typing and undo now work.'
fd,fn = tempfile.mkstemp(text=False)
s = g.toEncodedString(s)
os.write(fd,s)
os.close(fd)
f = open(fn,'rb')
s2 = f.read()
f.close()
assert s==s2
os.remove(fn)
print('deleted',fn)
.. @+node:ekr.20161117164321.746: *6* @@test round-trip-uAs for @shadow
root = p.firstChild()
uA = 'unknownAttributes'
tag = 'round-trip-u.uA'
ttag = 'round-trip-t.uA'
trace = False

if 0: # Set the uA's.
    for p2 in root.self_and_subtree_iter():
        p2.v.unknownAttributes = {tag: p2.h}
else: # Test the uA's.
    # The root is a special case.
    v = root.v
    assert hasattr(v,uA),'no v.uA for %s' % v
    assert getattr(v,uA),'empty v.uA for %s' % v
    for p2 in root.self_and_subtree_iter():
        v = p2.v
        assert hasattr(v,uA),'no v.uA for %s' % v
        a = getattr(v,uA)
        d = {tag: v.h}
        if trace: print(d)
        assert a == d, 'expected v.uA: "%s", got "%s"' % (d,a)
.. @+node:ekr.20161117164321.747: *7* @@shadow uA_test_shadow_file.py
@language python
@tabwidth -4
@others
.. @+node:ekr.20161117164321.748: *8* uA_test_shadow_file declarations
pass
pass
pass
.. @+node:ekr.20161117164321.749: *6* @@test round-trip-uAs for @thin
root = p.firstChild()
uA = 'unknownAttributes'
tag = 'round-trip-u.uA'
ttag = 'round-trip-t.uA'
trace = False

if 0: # Set the uA's.
    for p2 in root.self_and_subtree_iter():
        p2.v.unknownAttributes = {tag: p2.h}
else: # Test the uA's.
    # The root is a special case.
    v = root.v
    assert hasattr(v,uA),'no v.uA for %s' % v
    assert getattr(v,uA),'empty v.uA for %s' % v
    for p2 in root.self_and_subtree_iter():
        v = p2.v
        assert hasattr(v,uA),'no v.uA for %s' % v
        a = getattr(v,uA)
        d = {tag: v.h}
        if trace: print(d)
        assert a == d, 'expected v.uA: "%s", got "%s"' % (d,a)
.. @+node:ekr.20161117164321.750: *7* @@thin uA_test_file.py
@others
.. @+node:ekr.20161117164321.751: *8* child1
pass
.. @+node:ekr.20161117164321.752: *9* grandChild1
pass
.. @+node:ekr.20161117164321.753: *10* greatGrandChild1
pass
.. @+node:ekr.20161117164321.754: *5* New import tests
.. @+node:ekr.20161117164321.755: *6* @test skipToTheNextClassOrFunction (a class next)
import leo.core.leoImport as leoImport
ic = c.importCommands

s = '''\

def one():
    pass

import a
from . import a

@language python

d = {} # An interior comment.

# This is a comment.
# and another comment.
@aDecorator
class cl: # An interior comment
    def method(self):
        pass

def two():
    pass

'''

# tree = c.importCommands.pythonUnitTest(p,s=s,showTree=True)

expected = s.find('# This is a comment')
scanner = leoImport.PythonScanner(importCommands=ic,atAuto=False)
i = s.find('import a')
assert i > -1
i = scanner.skipToTheNextClassOrFunction(s,i,lastIndent=0)
assert i==expected,'expected %s, got %s %s' % (
    expected,i,repr(s[i:]))
.. @+node:ekr.20161117164321.756: *6* @test skipToTheNextClassOrFunction (a function next)
import leo.core.leoImport as leoImport
ic = c.importCommands

s = '''\

def one():
    pass

import a
from . import a

d = {}

# This is a comment.
@tabwith -4 # This looks like a comment.
# and another comment.
@aDecorator
def two():
    pass

'''

# tree = c.importCommands.pythonUnitTest(p,s=s,showTree=True)

expected = s.find('# This is a comment')
scanner = leoImport.PythonScanner(importCommands=ic,atAuto=False)
i = s.find('import a')
assert i > -1
i = scanner.skipToTheNextClassOrFunction(s,i,lastIndent=0)
assert i==expected,'expected %s, got %s %s' % (
    expected,i,repr(s[i:]))
.. @+node:ekr.20161117164321.757: *6* @test skipToTheNextClassOrFunction (nothing next)
import leo.core.leoImport as leoImport
ic = c.importCommands

s = '''\

def one():
    pass

import a
from . import a

d = {}

# This is a comment.
# and another comment.

@tabwidth -4

aList = ('a','b','def')

if __name__ == '__main__':
    pass

'''

lastLine = 'pass\n'
expected = s.find(lastLine) + len(lastLine) + 1
scanner = leoImport.PythonScanner(importCommands=ic,atAuto=False)
i = s.find('import a')
assert i > -1
i = scanner.skipToTheNextClassOrFunction(s,i,lastIndent=0)
assert i==expected,'expected %s, got %s %s' % (
    expected,i,repr(s[i:]))
.. @+node:ekr.20161117164321.758: *6* @test skipToTheNextClassOrFunction (indented def next)
import leo.core.leoImport as leoImport
ic = c.importCommands

s = '''\

def one():
    pass

import a
from . import a

if 0:
    def two():
        pass

if __name__ == '__main__':
    pass

'''

scanner = leoImport.PythonScanner(importCommands=ic,atAuto=False)
expected = i = s.find('import a')
assert i > -1
i = scanner.skipToTheNextClassOrFunction(s,i,lastIndent=0)
assert i==expected,'expected %s, got %s %s' % (
    expected,i,repr(s[i:]))
.. @+node:ekr.20161117164321.759: *5* Old Tests with @auto as the root
.. @+node:ekr.20161117164321.760: *6* @test goto-line-number @auto 1
# Not valid for external tests: uses @<file> node.
if not g.app.isExternalUnitTest:
    root = p.parent().parent()
    h = '@auto unittest/at-auto-line-number-test.py'
    target = g.findNodeAnywhere(c, h)
    assert target, 'no @auto node'
    p, n, found = c.gotoCommands.find_file_line(1, target)
    assert found, 'not found'
    assert n == 1, 'n: %s' % (n)
    assert p.h == 'at_auto_child', p.h
.. @+node:ekr.20161117164321.761: *6* @test goto-global-line @auto 2
# Not valid for external tests: uses @<file> node.
if not g.app.isExternalUnitTest:
    h = '@auto unittest/at-auto-line-number-test.py'
    root = g.findNodeAnywhere(c, h)
    assert root, 'no root'
    p, n, found = c.gotoCommands.find_file_line(20, root)
    assert not found, 'not found'
.. @+node:ekr.20161117164321.762: *6* @test goto-global-line @auto 3
# Not valid for external tests: uses @<file> node.
if not g.app.isExternalUnitTest:
    h = '@auto unittest/at-auto-line-number-test.py'
    root1 = g.findNodeAnywhere(c, h)
    assert root1
    assert root1.isAnyAtFileNode()
    fileName, isScript, lines, n, root = c.gotoCommands.setup_file(n=3, p=root1)
    assert fileName == h[6:], 'fileName: %s' % (fileName)
    assert root == root1, 'root: %s, root1: %s' % (root and root.h, root1 and root1.h)
    if 0:
        print('root:%s, isRaw:%s, n:%s, len(lines): %s' % (
            root and root.h, isRaw, n, len(lines)))
.. @+node:ekr.20161117164321.763: *6* @test goto-global-line @auto 4
# Not valid for external tests: uses @<file> node.
if not g.app.isExternalUnitTest:
    h = '@auto unittest/at-auto-line-number-test.py'
    root1 = g.findNodeAnywhere(c, h)
    assert root1
    assert root1.isAnyAtFileNode()
    scriptData = {'p': root1.copy(), 'lines': ['a', 'b', 'c']}
    fileName, lines2, p2, root2 = c.gotoCommands.setup_script(scriptData)
    assert fileName == h[6:], 'fileName'
    assert lines2 == scriptData.get('lines'), 'lines'
    assert p2 == root1, 'p'
    assert root2 == root1, 'root'
    if 0:
        print('root:%s, n:%s, len(lines): %s' % (
            root and root.h, n, len(lines)))
.. @+node:ekr.20161117164321.764: *6* @test goto-global-line @auto 5
# Not valid for external tests: uses @<file> node.
if not g.app.isExternalUnitTest:
    fn = '../test/at-auto-unit-test.py'
    root = g.findNodeAnywhere(c, '@auto %s' % (fn))
    assert root, 'no root'
    child1 = root.firstChild()
    assert child1, 'no child1'
    grand11 = child1.firstChild()
    assert grand11, 'no grand11'
    grand12 = grand11.next()
    assert grand12, 'no grand12'
    child2 = child1.next()
    assert child2, 'no child2'
    grand21 = child2.firstChild()
    assert grand21, 'no grand21'
    grand22 = grand21.next()
    assert grand22, 'no grand22'

    def oops(found, p2, node, n, n2):
        result = [' ']
        result.append('goto-global-line test failed at line %s' % (n))
        if not found:
            result.append('line %s not found', n)
        else:
            result.append('got node "%s", expected "%s"' % (p2.h, node.h))
            result.append('got offset %s, expected %s' % (n2, n))
        return '\n'.join(result)

    table = (
        # Use 1-based numbers externally.
        # find_file_line converts to zero-based numbers.
        (1, child1, 0),
        (2, grand11, 0),
        (3, grand11, 1),
        (4, grand12, 0),
        (5, grand12, 1),
        (6, child1, 2),
        (7, child2, 0),
        (8, grand21, 0),
        (9, grand21, 1),
        (10, grand22, 0),
        (11, grand22, 1),
        (12, root, 3),
    )
    # Test against actual lines of the file.
    path = g.os_path_finalize_join(g.app.loadDir, '..', 'test', fn)
    f = open(path, 'r'); s = f.read(); f.close()
    lines = g.splitLines(s) # The lines from the file.
    for n, node, index in table:
        p2, n2, found = c.gotoCommands.find_file_line(n, root)
            # n, the argument to find_file_line, is 1-based
            # n2, the returned index into p2.b, is zero-based
        n -= 1
            # Convert n to zero-based for the comparisons below.
        lines2 = g.splitLines(p2.b)
        if 0:
            print('%2d %s' % (n, repr(lines[n])))
        else:
            ok = lines2[n2].lstrip() == lines[n].lstrip()
            if not ok:
                i = 0
                for z in lines:
                    print('%2d %s' % (i, repr(z)))
                    i += 1
            if not ok:
                print('at line %s, index %s, node %s\ngot line %s\nexpected %s' % (
                    n, n2, p2.h, repr(lines2[n2].lstrip()), repr(lines[n].lstrip())))
            assert ok
.. @+node:ekr.20161117164321.765: *6* @test GoToLineNumber.find_root 1
# Not valid for external tests: uses @<file> node.
if not g.app.isExternalUnitTest:
    root = p.parent().parent()
    h = '@auto unittest/at-auto-line-number-test.py'
    target = g.findNodeAnywhere(c, h)
    assert target, 'no target'
    child = target.firstChild()
    assert child.h == 'at_auto_child', 'child.h'
    p, found = c.gotoCommands.find_root(child)
    assert p == target, 'p' #p and p.h
    assert found, 'not found'
.. @+node:ekr.20161117164321.766: *4* Cache tests
.. @+node:ekr.20161117164321.767: *5* @@test leoCache
# Disabled this test because it's best to open this file without caching.

import leo.core.leoCache as leoCache

cacher = leoCache.cacher(c)

assert cacher.test()
.. @+node:ekr.20161117164321.768: *4* Load tests for .leo files
@language python
@tabwidth -4

# These work well with the qttabs gui.
.. @+node:ekr.20161117164321.769: *5* @test test.leo
path = g.os_path_join(g.app.loadDir,"..","test","test.leo")
c.testManager.runLeoTest(path)
.. @+node:ekr.20161117164321.770: *5* @test leoDist.leo
path = g.os_path_join(g.app.loadDir,"..","dist","leoDist.leo")
c.testManager.runLeoTest(path)
.. @+node:ekr.20161117164321.771: *5* @test leoGuiPluginsRef.leo
path = g.os_path_join(g.app.loadDir,"..","plugins","leoGuiPluginsRef.leo")
c.testManager.runLeoTest(path)
.. @+node:ekr.20161117164321.772: *5* @test LeoPyRef.leo
path = g.os_path_join(g.app.loadDir,"..","core","LeoPyRef.leo")
c.testManager.runLeoTest(path)
.. @+node:ekr.20161117164321.773: *5* @test leoPluginsRef.leo
path = g.os_path_join(g.app.loadDir,"..","plugins","leoPluginsRef.leo")
c.testManager.runLeoTest(path)
.. @+node:ekr.20161117164321.774: *5* @test LeoDocs.leo
path = g.os_path_join(g.app.loadDir,"..","doc","LeoDocs.leo")
c.testManager.runLeoTest(path)
.. @+node:ekr.20161117164321.775: *5* @test minimalLeoFile.leo
path = g.os_path_join(g.app.loadDir,"..","test","unittest","minimalLeoFile.leo")
c.testManager.runLeoTest(path)
.. @+node:ekr.20161117164321.776: *5* @test minimalLeoFile2.leo
path = g.os_path_join(g.app.loadDir,"..","test","unittest","minimalLeoFile2.leo")
c.testManager.runLeoTest(path)
.. @+node:ekr.20161117164321.777: *5* @test minimalLeoFile3.leo
path = g.os_path_join(g.app.loadDir,"..","test","unittest","minimalLeoFile3.leo")
c.testManager.runLeoTest(path)
.. @+node:ekr.20161117164321.778: *4* slow tests
.. @+node:ekr.20161117164321.779: *5* @@test k.inverseCommandsDict is inverse of c.commandsDict
# c.commandsDict: keys are emacs command names, values are functions f.
# k.inverseCommandsDict: keys are f.__name__, values are emacs command names.

@others

d1 = c.commandsDict ; d2 = c.k.inverseCommandsDict

if 0:
    vals = d2.values() ; vals.sort()
    vals = [z for z in vals if z.startswith('contract')]
    g.pr('inverseCommandsDict.values()',vals)

keys1 = d1.keys() ; keys1.sort()
vals1 = d1.values()
vals1 = [f.__name__ for f in vals1]
vals1.sort()

keys2 = d2.keys() ; keys2.sort()
vals2 = d2.values(); vals2.sort()

if 0:
    g.pr(keys1,'\n\n')
    g.pr(vals2,'\n\n')
    g.pr(keys2,'\n\n')
    g.pr(vals1)

# g.trace(g.dictToString(c.k.abbreviationsDict))

abbrevDict = c.config.getAbbrevDict()

# Find @button and @command nodes in this file.
@others
buttonKeys = []
for p in c.allNodes_iter():
    h = p.h.strip().lower()
    for kind in ('@button','@command'):
        if h.startswith(kind):
            key1 = mungeKey(h,kind,substitute=False)
            if key1 not in buttonKeys:
                buttonKeys.append(key1)
            key = mungeKey(h,kind,substitute=True)
            if key not in buttonKeys:
                buttonKeys.append(key)

for z in g.app.config.atCommonButtonsList:
    h,junk = z
    key = mungeKey(h,'@button')
    # g.trace(key)
    if key not in buttonKeys:
        buttonKeys.append(key)

# g.pr('buttonKeys',buttonKeys)

for key in keys1:
    if key not in vals2 and key.lower() not in vals2:
        if (
            key.startswith('enter-') and key.endswith('-mode') or
            key.startswith('press-') and key.endswith('-button') or
            key.startswith('delete-') and key.endswith('-button') or
            key.startswith('nav-') and key.endswith('-menu')
        ):
            vals2.append(key)
        elif key in buttonKeys or key.lower() in buttonKeys:
            # List of buttons defined in this file, or in @settings tree.
            vals2.append(key)
        elif key.startswith('open-with-'):
            vals2.append(key)
        elif key in abbrevDict.keys():
            pass # g.trace('abbrev',key)
        else:
            assert False, '%s not in inverseCommandsDict.values()' % key

vals2.sort()
for val in vals2:
    if val not in keys1:
        assert False, '%s not in commandsDict.keys()' % (val)
.. @+node:ekr.20161117164321.780: *6* mungeKey
def mungeKey (h,kind,substitute=True):

    key = h[len(kind):].strip()
    i = key.find('@key')
    if i > -1: key = key[:i].strip()
    if substitute:
        key = key.replace(' ','-')
    # g.trace(key)
    return key


.. @+node:ekr.20161117164321.781: *5* @@test position > operator
# Disabled because very slow
aList = [z.copy() for z in c.all_positions()]

total = 0

for i in range(len(aList)):
    for j in range(0,i):
        assert aList[j] < aList[i],(i,j)
        total += 1

g.es('"%s": total tests: %s' % (p.h,total))
.. @+node:ekr.20161117164321.782: *4* @ignore from @mark-for-unit-tests-->importTests
.. @+node:ekr.20161117164321.783: *5* importAtFile
.. @+node:ekr.20161117164321.784: *6* dialog
openFileDialog
test\\unittest\\perfectImport\\formatter.py
.. @+node:ekr.20161117164321.785: *5* importAtRoot
.. @+node:ekr.20161117164321.786: *6* dialog
openFileDialog
test\\unittest\\perfectImport\\formatter.py
.. @+node:ekr.20161117164227.1: *4* Unit Tests for character-oriented scanners
.. @+node:ekr.20161202053956.1: *5* @test Importer.check
@first # -*- coding: utf-8 -*-

import leo.core.leoImport as leoImport
import leo.plugins.importers.linescanner as linescanner
import sys

ic = c.importCommands
runner = linescanner.Importer(ic,atAuto=True,language='python')
runner.root = p.copy()

g.app.unitTestDict ['expectedMismatchLine'] = 0

s1 = 'line Ä, 궯, 奠 end'
s2 = 'line Ä, 궯, end'

# Tracing Importer.check causes a UnicodeDecodeError.
ok = runner.check(s1=s1,s2=s2)

assert ok
.. @+node:ekr.20161130051250.1: *5* @test ic.RstScanner.removeBlankLinesTokens (rst)
# Important: at present only the RstScanner sets ignoreBlankLines == True
import leo.plugins.importers.leo_rst as leo_rst
sc = leo_rst.RstScanner(importCommands=c.importCommands,atAuto=True)
assert sc.ignoreBlankLines,'fail0'
def strip(tokens):
    '''Remove the line number item from all tokens.'''
    return [(kind,val) for kind,val,n in tokens]
table = (
    ('a\n\nb',          'a\nb'),
    ('a\n \t\nb',       'a\nb'),
    ('a\n \n\t\n\n\nb', 'a\nb'),
    ('a\nb\n',          'a\nb\n'),
)
for s,expected in table:
    tokens = sc.tokenize(s)
    s2 = ''.join([val for (kind,val,n) in tokens])
    assert s == s2,'fail1\nexpected:\n%s\ngot:\n%s' % (
        repr(s),repr(s2))
    # A: Remove tokens for blank lines from tokens.
    tokens2 = sc.removeBlankLinesTokens(tokens)
    # B: Remove blank lines first, then tokenize.
    lines = g.splitLines(s)
    lines2 = [z for z in lines if z.strip()]
    s2 = ''.join(lines2)
    tokens3 = sc.tokenize(s2)
    if 0:
        for kind,val,n in tokens3:
            print('%3s %7s %s' % (n,kind,repr(val)))
    # A and B should give the same result: just like abstract algebra diagrams.
    if 0:
        if strip(tokens2) != strip(tokens3):
            sc.compareTokens(tokens2,tokens3,trace=True)
    assert strip(tokens2) == strip(tokens3),'fail2\nexpected:\n%s\ngot:\n%s' % (
        strip(tokens3),strip(tokens2))
    # C: Removing blank tokens from already-compressed tokens should have no effect.
    tokens4 = sc.removeBlankLinesTokens(tokens3)
    assert tokens4 == tokens3,'fail3'
.. @+node:ekr.20161117164227.2: *5* @test ic.BaseScanner.insertIgnoreDirective
import leo.core.leoImport as leoImport
import leo.plugins.importers.basescanner as basescanner

def setup(p):
    while p.hasChildren():
        p.firstChild().doDelete()

importCommands = c.importCommands
bic = basescanner.BaseScanner(importCommands,atAuto=False,language='python')

try:
    setup(p)
    child = p.insertAsLastChild()
    child.h = 'child'
    bic.insertIgnoreDirective(parent=child)
    assert child.b.find('@ignore') == 0
finally:
    if 1:
        setup(p)
.. @+node:ekr.20161117164227.3: *5* @test ic.compareTokens: mismatched length
import leo.core.leoImport as leoImport
import leo.plugins.importers.basescanner as basescanner

ic = c.importCommands
bs = basescanner.BaseScanner(ic,atAuto=True,language='html')

<< define tokens >>

table = (
    (tokens11,tokens12),
    (tokens21,tokens22),
)

for tokens1,tokens2 in table:
    bs.compareTokens(tokens1,tokens2)
.. @+node:ekr.20161117164227.4: *6* << define tokens >>
@ 2011/11/10:

File "/usr/fetching/leo-editor/leo/core/leoImport.py", line 2074, in scanAndCompare
n1,n2,ok = self.compareTokens(tokens1,tokens2)
File "/usr/fetching/leo-editor/leo/core/leoImport.py", line 2094, in compareTokens
else:      kind1,val1 = 'eof','',n1
ValueError: too many values to unpack
@c

tokens11 = ()
tokens12 = (('id','abc',0),)

# Test similar situation, reversed.

tokens21 = (('id','abc',0),)
tokens22 = ()
.. @+node:ekr.20161117164227.5: *5* @test ic.createImportParent
files = ('x.h','x.cpp')

while p.hasChildren():
    p.firstChild().doDelete()

try:
    current = c.importCommands.createImportParent(c.p,files)
    assert current
    assert current.h == 'x'
    assert p.firstChild() == current
finally:
    while p.hasChildren():
        p.firstChild().doDelete()
    c.redraw()
.. @+node:ekr.20161117164227.6: *5* @test ic.HtmlScanner.adjust_class_ref
import leo.plugins.importers.html as html
ic = c.importCommands
hs = html.HtmlScanner(importCommands=ic,atAuto=True)

s = '''
<aTag>  @others
</aTag>
'''

# Avoid probems with representation of @others in scripts.
expected = '\n<aTag>\n@others\n</aTag>\n'

result = hs.adjust_class_ref(s)
assert result == expected,'expected...\n%s\ngot...\n%s' % (
    repr(expected),repr(result))
.. @+node:ekr.20161117164227.7: *5* @test ic.HtmlScanner.skipComment/Id/String
import leo.plugins.importers.html as html
ic = c.importCommands
hs = html.HtmlScanner(importCommands=ic,atAuto=True)

table = (
    # Yes, both single and double quotes are valid in html.
    (hs.skipComment, '<!-- comment --> after',  '<!-- comment -->'),
    (hs.skipComment, '<!-- a\nb --> after',     '<!-- a\nb -->'),
    (hs.skipId,      'abc>',                    'abc'),
    (hs.skipId,      'abc"',                    'abc'),
    (hs.skipId,      'abc<!--',                 'abc'),
    (hs.skipId,      'a.b-c9:d after',          'a.b-c9:d'),
    (hs.skipString,  '"a string" after',        '"a string"'),
    (hs.skipString,  "'a string2' after",       "'a string2'"),
    (hs.skipString,  "'a string<'>",            "'a string<'"),
    (hs.skipString,  "'a string>'<",            "'a string>'"),
)

for func,s,expected in table:
    
    i = func(s,0)
    result = s[0:i]
    assert result == expected,'expected %s got %s' % (
        repr(expected),repr(result))
.. @+node:ekr.20161117164227.8: *5* @test ic.reportMismatch
import leo.plugins.importers.leo_rst as leo_rst
ic = c.importCommands
scanner = leo_rst.RstScanner(importCommands=ic,atAuto=True)
scanner.root = p
s1 = ["abc","xyz",]
s2 = ["xyz",]
scanner.reportMismatch(s1,s2,1,1)
s1 = ["xyz",]
s2 = ["abc","xyz",]
scanner.reportMismatch(s1,s2,1,1)
.. @+node:ekr.20161117164227.9: *5* @test ic.skip...Token (HtmlScanner)
import leo.plugins.importers.html as html
ic = c.importCommands
scanner = html.HtmlScanner(importCommands=ic,atAuto=True)
tails = (
    '<whatever>',
    '+ abc', # don't concatenate with id or whitespace.
    '<!-- tail comment -->',
    '"tail string"',
) 
table = (
    (scanner.skipCommentToken,  '<!-- Test -->'),
    (scanner.skipIdToken,       'a_b-c.d:e'), # Valid in xml ids:  ".-:"
    # (scanner.skipNewlineToken,'\n'),
        # xmlScanner.skipNewlineToken throws exception (on purpose).
    (scanner.skipOtherToken,    '+'),
    (scanner.skipOtherToken,    '#'),
    (scanner.skipStringToken,   '"A string"'),
    (scanner.skipWsToken,       ' '),
)

# Special test for whitespace: Converts all runs of whitespace to a single blank.
if 0: # No longer does this.
    s = ' \n\t\t \n'
    i,result = scanner.skipWsToken(s+tails[0],0)
    assert i == len(s),'expected i==%s, got i==%s' % (len(s),i)
    expected = ' '
    assert result == expected,'expected...\n%s\ngot...\n%s' % (
        repr(expected),repr(result))
    
for f,s in table:
    for tail in tails:
        i,result = f(s+tail,0)
        expected = s
        assert i == len(s),'expected i==%s, got i==%s' % (len(s),i)
        assert result == expected,'expected...\n%s\ngot...\n%s' % (
            repr(expected),repr(result))
.. @+node:ekr.20161117164014.1: *5* @test ic.skip...Token (PythonScanner)
import leo.plugins.importers.python as python
ic = c.importCommands
scanner = python.PythonScanner(importCommands=ic,atAuto=True)
tails = (
    '+ abc', # don't concatenate with id or whitespace.
    '# tail comment',
    '"tail string"',
    "'tail string'",
)
table = (
    (scanner.skipCommentToken,  '# Test'),
    (scanner.skipIdToken,       'ab_c'),
    (scanner.skipNewlineToken,  '\n'),
    (scanner.skipOtherToken,    '+'),
    (scanner.skipOtherToken,    '#'),
    (scanner.skipStringToken,   '"A string"'),
    (scanner.skipWsToken,       ' '),
    (scanner.skipWsToken,       '\t '),
)
    
for f,s in table:
    for tail in tails:
        if f.__name__ == 'skipCommentToken':
            i,result = f(s,0)
        else:
            i,result = f(s+tail,0)
        expected = s
        assert i == len(s),'expected i==%s, got i==%s' % (len(s),i)
        assert result == expected,'expected...\n%s\ngot...\n%s' % (
            repr(expected),repr(result))
.. @+node:ekr.20161117164227.10: *5* @test ic.tokenize (HtmlScanner)
import leo.plugins.importers.html as html
ic = c.importCommands
hs = html.HtmlScanner(importCommands=ic,atAuto=True)
s = '''
<!-- a comment -->
<html "string">
Test.
</html>
'''
<< define expected >>
result = hs.tokenize(s)
if 1:
    assert result == expected,'expected...\n%s\ngot...\n%s' % (
        repr(expected),repr(result))
else:
    print(result)
    
@
AssertionError: expected...
[('ws', ' ', 0), ('comment', '<!-- a comment -->', 1), ('ws', ' ', 1), ('other', '<', 2), ('id', 'html', 2), ('ws', ' ',
 2), ('string', '"string"', 2), ('other', '>', 2), ('ws', ' ', 2), ('id', 'Test.', 3), ('ws', ' ', 3), ('other', '<', 4)
, ('other', '/', 4), ('id', 'html', 4), ('other', '>', 4), ('ws', ' ', 4)]
got...
[('nl', '\n', 0), ('comment', '<!-- a comment -->', 1), ('nl', '\n', 1), ('other', '<', 2), ('id', 'html', 2), ('ws', '
', 2), ('string', '"string"', 2), ('other', '>', 2), ('nl', '\n', 2), ('id', 'Test.', 3), ('nl', '\n', 3), ('other', '<'
, 4), ('other', '/', 4), ('id', 'html', 4), ('other', '>', 4), ('nl', '\n', 4)]

----------------------------------------------------------------------
Ran 1 test in 0.018s

FAILED (failures=1)

.. @+node:ekr.20161117164227.11: *6* << define expected >>
# expected = [
    # ('ws', ' ', 0),
    # ('comment', '<!-- a comment -->', 1),
    # ('ws', ' ', 1),
    # ('other', '<', 2), ('id', 'html', 2),
    # ('ws', ' ',2), ('string', '"string"', 2),
    # ('other', '>', 2),
    # ('ws', ' ', 2),
    # ('id', 'Test.', 3),
    # ('ws', ' ', 3),
    # ('other', '<', 4), ('other', '/', 4), ('id', 'html', 4), ('other', '>', 4),
    # ('ws', ' ', 4),
# ]

expected = [
    ('nl', '\n', 0),
    ('comment', '<!-- a comment -->', 1),
    ('nl', '\n', 1),
    ('other', '<', 2),
    ('id', 'html', 2),
    ('ws', ' ', 2),
    ('string', '"string"', 2),
    ('other', '>', 2),
    ('nl', '\n', 2),
    ('id', 'Test.', 3),
    ('nl', '\n', 3),
    ('other', '<', 4),
    ('other', '/', 4),
    ('id', 'html', 4),
    ('other', '>', 4),
    ('nl', '\n', 4),
]
.. @+node:ekr.20170127152303.1: *3* @test BaseColorizer
@language python
import imp
import leo.core.leoColorizer as leoColorizer
imp.reload(leoColorizer)
x = leoColorizer.BaseColorizer(c)
child = p.firstChild().firstChild()
x.updateSyntaxColorer(child)
assert x.language == 'html', x.language
.. @+node:ekr.20170127152638.1: *4* child
@language html
.. @+node:ekr.20170128142140.1: *5* grand-child
.. @+node:ekr.20170131112218.1: *3* @test filter.create_key_event
@first # -*- coding: utf-8 -*-
g.cls()
import leo.plugins.qt_events as qt_events
w = None
filter_ = qt_events.LeoQtEventFilter(c, w=w, tag='test')
aList = [
    'È',
    'a', 'A', '-',
    '<return>',
    '\n', 
    ' ', '\t',
]
event, tkKey = None, None
for shortcut in aList:
    ch = shortcut if len(shortcut) is 1 else ''
    event = filter_.create_key_event(event, c, w, ch, tkKey, shortcut)
    g.trace('%10r %r' % (shortcut, event))
        
.. @+node:ekr.20170206165145.1: *3* script: test demo.py
g.cls()
# c.frame.log.clearLog()
if c.isChanged(): c.save()
import imp
from leo.core.leoQt import QtGui
import leo.plugins.demo as demo
imp.reload(demo)
table = [
'''\
demo.delete_widgets()
demo.callout('Callout 1 centered')
demo.subtitle('This is subtitle 1')
''',
'''\
demo.delete_widgets()
demo.callout('Callout 2 (700, 200)', position=[700, 200])
demo.subtitle('This is subtitle 2')
''',
'''\
demo.delete_widgets()
demo.callout('Callout 3 (200, 300)', position=[200, 300])
demo.subtitle('This is subtitle 3')
''',
'''\
demo.delete_widgets()
demo.callout('Callout 4 (center, 200)', position=['center', 200])
demo.subtitle('This is a much much longer subtitle 4')
''',
'''\
demo.delete_widgets()
demo.callout('Callout 5 (700, center)', position=[700, 'center'])
demo.subtitle('Short 5')
''',
'''\
demo.delete_widgets()
demo.next()
''',
]
color = QtGui.QColor('lightblue')
sub_color = QtGui.QColor('mistyrose')
demo = demo.Demo(c, color=color, subtitle_color=sub_color, trace=False)
demo.delete_widgets()
demo.start(script_list = table)
.. @+node:ekr.20170317101032.1: *3* test g.unCamel
g.cls()

table = (
    'abcXyz',
    'AbcXyz',
    'abcXyzW',
)
for s in table:
    print(s)
    g.printList(g.unCamel(s))
.. @+node:ekr.20170107222248.1: ** Pyzo: do not delete
@language python
.. @+node:ekr.20170107211202.1: *3* COPY ../external/pyzo/highlighter.py
@first # -*- coding: utf-8 -*-
<< pyzo copyright >>
<< highlighter imports >>
@others
@language python
@tabwidth -4
.. @+node:ekr.20170108171824.1: *4* << pyzo copyright >>
# Copyright (C) 2016, the Pyzo development team
#
# Pyzo is distributed under the terms of the (new) BSD License.
# The full license can be found in 'license.txt'.
.. @+node:ekr.20170107222425.1: *4* << highlighter imports >>
from leo.core.leoQt import QtCore, QtGui
import leo.core.leoGlobals as g
ustr = g.ustr
from .parsers import BlockState
from .style import StyleFormat # New.
import re
.. @+node:ekr.20170107211216.3: *4* class BlockData
class BlockData(QtGui.QTextBlockUserData):
    """ Class to represent the data for a block.
    """
    @others
# The highlighter should be part of the base class, because 
# some extensions rely on them (e.g. the indent guuides).
.. @+node:ekr.20170107211216.4: *5* __init__
def __init__(self):
    QtGui.QTextBlockUserData.__init__(self)
    self.indentation = None
    self.fullUnderlineFormat = None
    self.tokens = []
.. @+node:ekr.20170107211216.5: *4* class PyzoHighlighter
class PyzoHighlighter(QtGui.QSyntaxHighlighter):
    @others
.. @+node:ekr.20170107211216.6: *5* pyzo_h.__init__ & helpers

# Don't use codeEditor at all.

def __init__(self, parser, *args):
    # Set these *before* initing the base class.
    # self._codeEditor = codeEditor
    self.parser = parser
    self.colorer = None
        # To disable some code in qtew.setAllText.
    d = self.defineStyles()
    self.initStyles(d)
    assert self.style_d
    QtGui.QSyntaxHighlighter.__init__(self, *args)
        # Generates call to rehighlight.
.. @+node:ekr.20170112103148.1: *6* h.define_styles (new)
def defineStyles(self):
    '''Set self.style_format_d.'''
    base03  = "#002b36"
    base02  = "#073642"
    base01  = "#586e75"
    base00  = "#657b83"
    base0   = "#839496"
    base1   = "#93a1a1"
    base2   = "#eee8d5"
    base3   = "#fdf6e3"
    yellow  = "#b58900"
    orange  = "#cb4b16"
    red     = "#dc322f"
    magenta = "#d33682"
    violet  = "#6c71c4"
    blue    = "#268bd2"
    cyan    = "#2aa198"
    green   = "#859900"
    light = True
    if light: # Light background.
        #back1, back2, back3 = base3, base2, base1 # real solarised
        back1, back2, back3 = "#fff", base2, base1 # crispier
        fore1, fore2, fore3, fore4 = base00, base01, base02, base03
    else:
        back1, back2, back3 = base03, base02, base01
        fore1, fore2, fore3, fore4 = base0, base1, base2, base3
    S = {}
    plain     = "fore:%s, bold:no,  italic:no,  underline:no"
    bold      = "fore:%s, bold:yes, italic:no,  underline:no"
    dotted    = "fore:%s, bold:no,  italic:no,  underline:dotted"
    italic    = "fore:%s, bold:no,  italic:yes, underline:no"
    solid     = "linestyle:solid, fore:%s"
    underline = "fore:%s, bold:yes, italic:no,  underline:full"
    S["Editor.text"] = "back:%s, fore:%s" % (back1, fore1)
    S["Syntax.text"] = "back:%s, fore:%s" % (back1, fore1)
        ### Added: ekr
    S['Syntax.identifier'] = plain % fore1
    S["Syntax.nonidentifier"] = plain % fore2
    S["Syntax.keyword"] = bold % fore2
    #
    S["Syntax.functionname"] = bold % fore3
    S["Syntax.classname"] = bold % orange
    # EKR
    S["Syntax.openparen"] = plain % cyan
    S["Syntax.closeparen"] = plain % cyan
    #
    S["Syntax.string"] = plain % violet
    S["Syntax.unterminatedstring"] = dotted % violet
    S["Syntax.python.multilinestring"] = plain % blue
    #
    S["Syntax.number"] = plain % cyan
    S["Syntax.comment"] = plain % yellow
    S["Syntax.todocomment"] = italic % magenta
    S["Syntax.python.cellcomment"] = underline % yellow
    #
    S["Editor.Long line indicator"] = solid % back2
    S["Editor.Highlight current line"] = "back:%s" % back2
    S["Editor.Indentation guides"] = solid % back2
    S["Editor.Line numbers"] = "back:%s, fore:%s" % (back2, back3)
    # Normalize all keys.
    d = {}
    for key in S:
        normKey = key.replace(' ', '').lower()
        d[normKey] = S.get(key)
    return d
.. @+node:ekr.20170112101149.4: *6* h.initStyles (new)


def initStyles(self, d):

    # Set style elements. Keys have already been normalized.
    self.style_d = {}
    for key in d:
        self.style_d[key] = StyleFormat(format=d[key])
    # Notify that style changed.
    # adopt a lazy approach to make loading quicker.
    if False: ### self.isVisible(): ###
        callLater(self.styleChanged.emit)
        self.__styleChangedPending = False
    else:
        self.__styleChangedPending = True
.. @+node:ekr.20170107211216.7: *5* pyzo_h.getCurrentBlockUserData
n_block_data = 0

def getCurrentBlockUserData(self):
    """ getCurrentBlockUserData()
    
    Gets the BlockData object. Creates one if necesary.
    
    """
    self.n_block_data += 1
    bd = self.currentBlockUserData()
    if not isinstance(bd, BlockData):
        bd = BlockData()
        self.setCurrentBlockUserData(bd)
    return bd
.. @+node:ekr.20170107211216.8: *5* pyzo_h.highlightBlock
n_calls = 0
n_tokens = 0
lws_pattern = re.compile(r'(\s*)')

def highlightBlock(self, line): 
    """ highlightBlock(line)
    
    This method is automatically called when a line must be 
    re-highlighted.
    
    If the code editor has an active parser. This method will use
    it to perform syntax highlighting. If not, it will only 
    check out the indentation.
    
    """
    trace = False and not g.unitTesting
    self.n_calls += 1
    # Make sure this is a Unicode Python string
    line = ustr(line)
    # Get previous state
    previousState = self.previousBlockState()
    # Get parser
    parser = self.parser
    fullLineFormat = None
    tokens = []
    if parser:
        self.setCurrentBlockState(0)
        tokens = list(parser.parseLine(line, previousState))
        self.n_tokens += len(tokens)
        # g.trace(len(tokens), 'tokens', tokens)
        for token in tokens :
            # Handle block state
            if isinstance(token, BlockState):
                self.setCurrentBlockState(token.state)
                if trace: g.trace('block state')
            else:
                # Get format
                normKey = token.name.replace(' ', '').lower()
                styleFormat = self.style_d.get(normKey)
                if styleFormat:
                    # Set format
                    # if trace: g.trace(token.name,charFormat)
                    charFormat = styleFormat.textCharFormat
                    self.setFormat(token.start,token.end-token.start,charFormat)
                    fullLineFormat = styleFormat
                else:
                    g.trace('no format', repr(token.name))
    # Get user data
    bd = self.getCurrentBlockUserData()
    # Store token list for future use (e.g. brace matching)
    bd.tokens = tokens
    # Handle underlines
    bd.fullUnderlineFormat = fullLineFormat
    # Get the indentation setting of the editors
    indentUsingSpaces = True ### self._codeEditor.indentUsingSpaces()
    ### leadingWhitespace=line[:len(line)-len(line.lstrip())]
    m = self.lws_pattern.match(line)
    leadingWhitespace = m and m.group(1) or ''
    if 1:
        bd.indentation = len(leadingWhitespace)
    elif '\t' in leadingWhitespace and ' ' in leadingWhitespace:
        #Mixed whitespace
        bd.indentation = 0
        format=QtGui.QTextCharFormat()
        format.setUnderlineStyle(QtGui.QTextCharFormat.SpellCheckUnderline)
        format.setUnderlineColor(QtCore.Qt.red)
        format.setToolTip('Mixed tabs and spaces')
        self.setFormat(0,len(leadingWhitespace),format)
    elif (
        ('\t' in leadingWhitespace and indentUsingSpaces) or
        (' ' in leadingWhitespace and not indentUsingSpaces)
    ):
        # Whitespace differs from document setting
        bd.indentation = 0
        format=QtGui.QTextCharFormat()
        format.setUnderlineStyle(QtGui.QTextCharFormat.SpellCheckUnderline)
        format.setUnderlineColor(QtCore.Qt.blue)
        format.setToolTip('Whitespace differs from document setting')
        self.setFormat(0,len(leadingWhitespace),format)
    else:
        # Store info for indentation guides
        # amount of tabs or spaces
        bd.indentation = len(leadingWhitespace)
.. @+node:ekr.20170108091854.1: *5* pyzo_h.rehighlight (new)
def rehighlight(self, p=None):
    '''Leo override, allowing the 'p' keyword arg.'''
    trace = True and not g.unitTesting # and p and len(p.b) > 1000
    self.n_calls = self.n_block_data = self.n_tokens = 0
    self.parser.n_parse = 0
    if trace: g.trace('(pyzo) =====', p and p.h)
    QtGui.QSyntaxHighlighter.rehighlight(self)
    if trace:
        g.trace('(pyzo_h) -----',
            self.n_calls,
            self.n_block_data,
            self.parser.n_parse,
            self.n_tokens,
            p and p.h)
.. @+node:ekr.20170108044304.1: *3* COPY ../external/pyzo/base.py
@first # -*- coding: utf-8 -*-
"""The base code editor class."""
<< pyzo copyright >>
<< base.py imports >>
@others
@language python
@tabwidth -4
.. @+node:ekr.20170108171834.1: *4* << pyzo copyright >>
# Copyright (C) 2016, the Pyzo development team
#
# Pyzo is distributed under the terms of the (new) BSD License.
# The full license can be found in 'license.txt'.
.. @+node:ekr.20170108044517.1: *4* << base.py imports >>
import leo.core.leoGlobals as g
ustr = g.ustr
# from .qt import QtGui, QtCore, QtWidgets
from leo.core.leoQt import QtCore, QtGui, QtWidgets, isQt5
if isQt5:
    from PyQt5.QtCore import pyqtSignal as Signal
else:
    from PyQt4.QtCore import pyqtSignal as Signal
from .misc import DEFAULT_OPTION_NAME, DEFAULT_OPTION_NONE, ce_option
from .misc import callLater # , ustr
from .manager import Manager
from .highlighter import PyzoHighlighter ### Highlighter
from .style import StyleElementDescription, StyleFormat
.. @+node:ekr.20170108045413.3: *4* class CodeEditorBase
class CodeEditorBase(QtWidgets.QPlainTextEdit):
    """ The base code editor class. Implements some basic features required
    by the extensions.
    
    """
    # Style element for default text and editor background
    _styleElements = [
        ('Editor.text',
         'The style of the default text. ' + 
         'One can set the background color here.',
         'fore:#000,back:#fff',
        )
    ]
    # Signal emitted after style has changed
    styleChanged = Signal()
    
    # Signal emitted after font (or font size) has changed
    fontChanged = Signal()
    
    # Signal to indicate a change in breakpoints. Only emitted if the
    # appropriate extension is in use
    breakPointsChanged = Signal(object)
    
    @others
.. @+node:ekr.20170108045413.4: *5* ed.__init__ (sets solarized colors)
def __init__(self,*args, **kwds):
    super(CodeEditorBase, self).__init__(*args)
    # Set font (always monospace)
    self.__zoom = 0
    self.setFont()
    # Create highlighter class 
    self.__highlighter = PyzoHighlighter(self, self.document())
    # Set some document options
    option = self.document().defaultTextOption()
    option.setFlags(
        option.flags() |
        option.IncludeTrailingSpaces |
        option.AddSpaceForLineAndParagraphSeparators
    )
    self.document().setDefaultTextOption(option)
    # When the cursor position changes, invoke an update, so that
    # the hihghlighting etc will work
    self.cursorPositionChanged.connect(self.viewport().update) 
    # Init styles to default values
    self.leo_style = {}
    for element in self.getStyleElementDescriptions():
        self.leo_style[element.key] = element.defaultFormat
    # Connect style update
    self.styleChanged.connect(self.__afterSetStyle)
    self.__styleChangedPending = False
    # Init margins
    self._leftmargins = []
    # Init options now. 
    # NOTE TO PEOPLE DEVELOPING EXTENSIONS:
    # If an extension has an __init__ in which it first calls the 
    # super().__init__, this __initOptions() function will be called, 
    # while the extension's init is not yet finished.        
    self.__initOptions(kwds)
    # Define colors from Solarized theme
    # NOTE TO PEOPLE WANTING CUSTOM COLORS: ignore this and check the
    # commented lines near the bottom of this method.
    base03  = "#002b36"
    base02  = "#073642"
    base01  = "#586e75"
    base00  = "#657b83"
    base0   = "#839496"
    base1   = "#93a1a1"
    base2   = "#eee8d5"
    base3   = "#fdf6e3"
    yellow  = "#b58900"
    orange  = "#cb4b16"
    red     = "#dc322f"
    magenta = "#d33682"
    violet  = "#6c71c4"
    blue    = "#268bd2"
    cyan    = "#2aa198"
    green   = "#859900"
    if True: # Light vs dark
        #back1, back2, back3 = base3, base2, base1 # real solarised
        back1, back2, back3 = "#fff", base2, base1 # crispier
        fore1, fore2, fore3, fore4 = base00, base01, base02, base03
    else:
        back1, back2, back3 = base03, base02, base01
        fore1, fore2, fore3, fore4 = base0, base1, base2, base3
    test_numbers  = 90 + 0000 + 1
    # todo: proper testing of syntax style
    # Define style using "Solarized" colors
    S  = {}
    S["Editor.text"] = "back:%s, fore:%s" % (back1, fore1)
    S["Syntax.text"] = "back:%s, fore:%s" % (back1, fore1)
        ### Added: ekr
    S['Syntax.identifier'] = "fore:%s, bold:no, italic:no, underline:no" % fore1
    S["Syntax.nonidentifier"] = "fore:%s, bold:no, italic:no, underline:no" % fore2
    S["Syntax.keyword"] = "fore:%s, bold:yes, italic:no, underline:no" % fore2
    #
    S["Syntax.functionname"] = "fore:%s, bold:yes, italic:no, underline:no" % fore3
    S["Syntax.classname"] = "fore:%s, bold:yes, italic:no, underline:no" % orange
    # EKR
    S["Syntax.openparen"] = "fore:%s, bold:no, italic:no, underline:no" % cyan
    S["Syntax.closeparen"] = "fore:%s, bold:no, italic:no, underline:no" % cyan
    #
    S["Syntax.string"] = "fore:%s, bold:no, italic:no, underline:no" % violet
    S["Syntax.unterminatedstring"] = "fore:%s, bold:no, italic:no, underline:dotted" % violet
    S["Syntax.python.multilinestring"] = "fore:%s, bold:no, italic:no, underline:no" % blue
    #
    S["Syntax.number"] = "fore:%s, bold:no, italic:no, underline:no" % cyan
    S["Syntax.comment"] ="fore:%s, bold:no, italic:no, underline:no" % yellow
    S["Syntax.todocomment"] = "fore:%s, bold:no, italic:yes, underline:no" % magenta
    
    S["Syntax.python.cellcomment"] = "fore:%s, bold:yes, italic:no, underline:full" % yellow
    #
    S["Editor.Long line indicator"] = "linestyle:solid, fore:%s" % back2
    S["Editor.Highlight current line"] = "back:%s" % back2
    S["Editor.Indentation guides"] = "linestyle:solid, fore:%s" % back2
    S["Editor.Line numbers"] = "back:%s, fore:%s" % (back2, back3)
    
    # Define style using html color names. All 140 legal HTML colour
    # names can be used (in addition to HEX codes). A full list of
    # recognized colour names is available e.g. here
    # http://www.html-color-names.com/color-chart.php
        #         S  = {}
        #         S["Editor.text"] = "back: white, fore: black"
        #         S['Syntax.identifier'] = "fore: black, bold:no, italic:no, underline:no"
        #         S["Syntax.nonidentifier"] = "fore: blue, bold:no, italic:no, underline:no"
        #         S["Syntax.keyword"] = "fore: blue, bold:yes, italic:no, underline:no"
        #         S["Syntax.functionname"] = "fore: black, bold:yes, italic:no, underline:no"
        #         S["Syntax.classname"] = "fore: magenta, bold:yes, italic:no, underline:no"
        #         S["Syntax.string"] = "fore: red, bold:no, italic:no, underline:no"
        #         S["Syntax.unterminatedstring"] = "fore: red, bold:no, italic:no, underline:dotted"
        #         S["Syntax.python.multilinestring"] = "fore: red, bold:no, italic:no, underline:no"
        #         S["Syntax.number"] = "fore: dark orange, bold:no, italic:no, underline:no"
        #         S["Syntax.comment"] ="fore: green, bold:no, italic:yes, underline:no"
        #         S["Syntax.todocomment"] = "fore: magenta, bold:no, italic:yes, underline:no"
        #         S["Syntax.python.cellcomment"] = "fore: green, bold:yes, italic:no, underline:full"
        #         S["Editor.Long line indicator"] = "linestyle:solid, fore: dark grey"
        #         S["Editor.Highlight current line"] = "back: light grey"
        #         S["Editor.Indentation guides"] = "linestyle:solid, fore: light grey"
        #         S["Editor.Line numbers"] = "back: light grey, fore: black"
    
    # Apply style
    # g.trace('(CodeEditorBase)','*'*20) ; g.printDict(S)
    self.setStyle(S, force=True) # EKR
.. @+node:ekr.20170108045413.5: *5* _setHighlighter
def _setHighlighter(self, highlighterClass):
    self.__highlighter = highlighterClass(self, self.document())
.. @+node:ekr.20170108092633.1: *5* Option setters/getters
.. @+node:ekr.20170108045413.6: *6* ed.__getOptionSetters
def __getOptionSetters(self):
    """ Get a dict that maps (lowercase) option names to the setter
    methods.
    """
    # Get all names that can be options
    allNames = set(dir(self))
    nativeNames = set(dir(QtWidgets.QPlainTextEdit))
    names = allNames.difference(nativeNames)
    # Init dict of setter members
    setters = {}
    for name in names:
        # Get name without set
        if name.lower().startswith('set'):
            name = name[3:]
        # Get setter and getter name
        name_set = 'set' + name[0].upper() + name[1:]
        name_get = name[0].lower() + name[1:]
        # Check if both present
        if not (name_set in names and name_get in names):
            continue
        # Get members
        member_set = getattr(self, name_set)
        member_get = getattr(self, name_get)
        # Check if option decorator was used and get default value
        for member in [member_set, member_get]:
            if hasattr(member, DEFAULT_OPTION_NAME):
                defaultValue = member.__dict__[DEFAULT_OPTION_NAME]
                break
        else:
            continue
        # Set default on both
        member_set.__dict__[DEFAULT_OPTION_NAME] = defaultValue
        member_get.__dict__[DEFAULT_OPTION_NAME] = defaultValue
        # Add to list
        setters[name.lower()] = member_set
    # Done
    if 0: ###
        g.trace()
        # g.printDict(setters)
        g.printList(['%20s:%s' % (z, setters.get(z).__name__)
            for z in sorted(setters)])
    return setters
.. @+node:ekr.20170108045413.7: *6* ed.__setOptions
def __setOptions(self, setters, options):
    """ Sets the options, given the list-of-tuples methods and an
    options dict.
    """
    # List of invalid keys
    invalidKeys = []
    # Set options
    for key1 in options:
        key2 = key1.lower()
        # Allow using the setter name
        if key2.startswith('set'):
            key2 = key2[3:]
        # Check if exists. If so, call!
        if key2 in setters:
            fun = setters[key2]
            val = options[key1]
            fun(val)
        else:
            invalidKeys.append(key1)
    # Check if invalid keys were given
    if invalidKeys:
        print("Warning, invalid options given: " + ', '.join(invalidKeys))
.. @+node:ekr.20170108045413.8: *6* __initOptions
def __initOptions(self, options=None):
    """ Init the options with their default values.
    Also applies the docstrings of one to the other.
    """
    # Make options an empty dict if not given
    if not options:
        options = {}
    # Get setters
    setters = self.__getOptionSetters()
    # Set default value
    for member_set in setters.values():
        defaultVal = member_set.__dict__[DEFAULT_OPTION_NAME]
        if defaultVal != DEFAULT_OPTION_NONE:
            try:
                member_set(defaultVal)
            except Exception as why:
                print('Error initing option ', member_set.__name__)
    # Also set using given opions?
    if options:
        self.__setOptions(setters, options)
.. @+node:ekr.20170108045413.9: *6* ed.setOptions
def setOptions(self, options=None, **kwargs):
    """ setOptions(options=None, **kwargs)
    
    Set the code editor options (e.g. highlightCurrentLine) using
    a dict-like object, or using keyword arguments (options given
    in the latter overrule opions in the first).
    
    The keys in the dict are case insensitive and one can use the
    option's setter or getter name.
    
    """
    # Process options
    if options:
        D = {}            
        for key in options:
            D[key] = options[key]
        D.update(kwargs)
    else:
        D = kwargs
    # Get setters
    setters = self.__getOptionSetters()
    # Go
    self.__setOptions(setters, D)
.. @+node:ekr.20170108092733.1: *5* Font settings
.. @+node:ekr.20170108045413.10: *6* setFont
def setFont(self, font=None):
    """ setFont(font=None)
    
    Set the font for the editor. Should be a monospace font. If not,
    Qt will select the best matching monospace font.
    
    """
    defaultFont = Manager.defaultFont()
    # Get font object
    if font is None:
        font = defaultFont
    elif isinstance(font, QtGui.QFont):
        pass
    elif isinstance(font, str):
        font = QtGui.QFont(font)
    else:
        raise ValueError("setFont accepts None, QFont or string.")
    # Hint Qt that it should be monospace
    font.setStyleHint(font.TypeWriter, font.PreferDefault)
    # Get family, fall back to default if qt could not produce monospace
    fontInfo = QtGui.QFontInfo(font)
    if fontInfo.fixedPitch():
        family = fontInfo.family() 
    else:
        family = defaultFont.family()
    # Get size: default size + zoom
    size = defaultFont.pointSize() + self.__zoom
    # Create font instance
    font = QtGui.QFont(family, size)
    # Set, emit and return
    QtWidgets.QPlainTextEdit.setFont(self, font)
    self.fontChanged.emit()
    return font
.. @+node:ekr.20170108045413.11: *6* setZoom
def setZoom(self, zoom):
    """ setZoom(zoom)
    
    Set the zooming of the document. The font size is always the default
    font size + the zoom factor.
    
    The final zoom is returned, this may not be the same as the given
    zoom factor if the given factor is too small.
    
    """
    # Set zoom (limit such that final pointSize >= 1)
    size = Manager.defaultFont().pointSize()
    self.__zoom = int(max(1-size,zoom))
    # Set font
    self.setFont(self.fontInfo().family())
    # Return zoom
    return self.__zoom
.. @+node:ekr.20170108092658.1: *5* Syntax styling
.. @+node:ekr.20170108045413.12: *6* getStyleElementDescriptions
@classmethod
def getStyleElementDescriptions(cls):
    """ getStyleElementDescriptions()
    
    This classmethod returns a list of the StyleElementDescription 
    instances used by this class. This includes the descriptions for
    the syntax highlighting of all parsers.
    
    """
    # g.trace('-'*20, g.callers())
    # Collect members by walking the class bases
    elements = []
    def collectElements(cls, iter=1):
        # Valid class?
        if cls is object or cls is QtWidgets.QPlainTextEdit:
            return
        # Check members
        if hasattr(cls, '_styleElements'):
            for element in cls._styleElements:
                elements.append(element)
        # Recurse
        for c in cls.__bases__:
            collectElements(c, iter+1)
    collectElements(cls)
    # Make style element descriptions
    # (Use a dict to ensure there are no duplicate keys)
    elements2 = {}
    for element in elements:
        # Check
        if isinstance(element, StyleElementDescription):
            pass
        elif isinstance(element, tuple):
            element = StyleElementDescription(*element)
        else:
            print('Warning: invalid element: ' + repr(element))
        # Store using the name as a key to prevent duplicates
        elements2[element.key] = element
    # Done
    # g.trace() ; g.printList(list(elements2.values()))
    return list(elements2.values())
.. @+node:ekr.20170108045413.13: *6* ed.getStyleElementFormat
def getStyleElementFormat(self, name):
    """ getStyleElementFormat(name)
    
    Get the style format for the style element corresponding with
    the given name. The name is case insensitive and invariant to
    the use of spaces.
    
    """
    normKey = name.replace(' ','').lower()
    try:
        return self.leo_style[normKey]
    except KeyError:
        g.trace(normKey)
        raise KeyError('Not a known style element name: "%s".' % normKey)
.. @+node:ekr.20170108045413.14: *6* ed.setStyle
def setStyle(self, style=None, force=False, **kwargs):
    """ setStyle(style=None, **kwargs)
    
    Updates the formatting per style element. 
    
    The style consists of a dictionary that maps style names to
    style formats. The style names are case insensitive and invariant 
    to the use of spaces.
    
    For convenience, keyword arguments may also be used. In this case,
    underscores are interpreted as dots.
    
    This function can also be called without arguments to force the 
    editor to restyle (and rehighlight) itself.
    
    Use getStyleElementDescriptions() to get information about the
    available styles and their default values.
    
    Examples
    --------
    # To make the classname in underline, but keep the color and boldness:
    setStyle(syntax_classname='underline') 
    # To set all values for function names:
    setStyle(syntax_functionname='#883,bold:no,italic:no') 
    # To set line number and indent guides colors
    setStyle({  'editor.LineNumbers':'fore:#000,back:#777', 
                'editor.indentationGuides':'#f88' })
    
    """
    # Combine user input
    # def normkey(key):
        # return key.replace(' ', '').lower()
    g.trace('=====', 'force', force) #, '\n', g.callers())
    D = {}
    if style:
        for key in style:
            normKey = key.replace(' ', '').lower()
            # g.trace(key, '->', normKey)
            ### D[key] = style[key]
            D[normKey] = style[key]
    if True:
        for key in kwargs:
            key2 = key.replace('_', '.')
            D[key2] = kwargs[key]
    # List of given invalid style element names
    invalidKeys = []
    # g.printList(sorted(D.keys()))
    # Set style elements
    for key in D:
        normKey = key.replace(' ', '').lower()
        if force:
            self.leo_style[normKey] = StyleFormat(format=D[normKey])
        elif normKey in self.leo_style:
            ### self.leo_style[normKey].update(D[key])
            self.leo_style[normKey].update(D[normKey]) # EKR
        else:
            invalidKeys.append(key)
    # Give warning for invalid keys
    if invalidKeys:
        print("Warning, invalid style names given: \n" + 
            '\n'.join(sorted(invalidKeys)))
    # Notify that style changed, adopt a lazy approach to make loading
    # quicker.
    if True: ### self.isVisible(): ###
        callLater(self.styleChanged.emit)
        self.__styleChangedPending = False
    else:
        self.__styleChangedPending = True
.. @+node:ekr.20170108045413.15: *6* ed.showEvent
def showEvent(self, event):
    super(CodeEditorBase, self).showEvent(event)
    # Does the style need updating?
    if self.__styleChangedPending:
        callLater(self.styleChanged.emit)
        self.__styleChangedPending = False
.. @+node:ekr.20170108045413.16: *6* ed.__afterSetStyle
def __afterSetStyle(self):
    """ _afterSetStyle()
    
    Callback after the style has been set.
    """
    # Set text style using editor style sheet
    format = self.getStyleElementFormat('editor.text')
    ss = 'QPlainTextEdit{ color:%s; background-color:%s; }' %  (
        format['fore'], format['back'])
    self.setStyleSheet(ss)
    # Make sure the style is applied
    self.viewport().update()
    # Re-highlight
    callLater(self.__highlighter.rehighlight)
.. @+node:ekr.20170108092857.1: *5* Basic options
.. @+node:ekr.20170108045413.17: *6* indentWidth
@ce_option(4)
def indentWidth(self):
    """ Get the width of a tab character, and also the amount of spaces
    to use for indentation when indentUsingSpaces() is True.
    """
    return self.__indentWidth
.. @+node:ekr.20170108045413.18: *6* setIndentWidth
def setIndentWidth(self, value):
    value = int(value)
    if value<=0:
        raise ValueError("indentWidth must be >0")
    self.__indentWidth = value
    self.setTabStopWidth(self.fontMetrics().width('i'*self.__indentWidth))
.. @+node:ekr.20170108045413.19: *6* indentUsingSpaces
@ce_option(False)
def indentUsingSpaces(self):
    """Get whether to use spaces (if True) or tabs (if False) to indent
    when the tab key is pressed
    """
    return self.__indentUsingSpaces
.. @+node:ekr.20170108045413.20: *6* setIndentUsingSpaces
def setIndentUsingSpaces(self, value):
    self.__indentUsingSpaces = bool(value)
    self.__highlighter.rehighlight()
.. @+node:ekr.20170108092912.1: *5* Misc
.. @+node:ekr.20170108045413.21: *6* gotoLine
def gotoLine(self, lineNumber):
    """ gotoLine(lineNumber)
    
    Move the cursor to the block given by the line number 
    (first line is number 1) and show that line.
    
    """
    return self.gotoBlock(lineNumber-1)
.. @+node:ekr.20170108045413.22: *6* gotoBlock
def gotoBlock(self, blockNumber):
    """ gotoBlock(blockNumber)
    
    Move the cursor to the block given by the block number 
    (first block is number 0) and show that line.
    
    """
    # Two implementations. I know that the latter works, so lets
    # just use that.
    
    cursor = self.textCursor()
    #block = self.document().findBlockByNumber( blockNumber )
    #cursor.setPosition(block.position())
    cursor.movePosition(cursor.Start) # move to begin of the document
    cursor.movePosition(cursor.NextBlock,n=blockNumber) # n blocks down
    
    try:
        self.setTextCursor(cursor)
    except Exception:
        pass # File is smaller then the caller thought

    # TODO make this user configurable (setting relativeMargin to anything above
    # 0.5 will cause cursor to center on each move)
    relativeMargin = 0.2    # 20% margin on both sides of the window
    margin = self.height() * relativeMargin
    cursorRect = self.cursorRect(cursor)
    if cursorRect.top() < margin or cursorRect.bottom() + margin > self.height():
        self.centerCursor()
.. @+node:ekr.20170108045413.23: *6* doForSelectedBlocks
def doForSelectedBlocks(self, function):
    """ doForSelectedBlocks(function)
    
    Call the given function(cursor) for all blocks in the current selection
    A block is considered to be in the current selection if a part of it is in
    the current selection 
    
    The supplied cursor will be located at the beginning of each block. This
    cursor may be modified by the function as required
    
    """
    
    #Note: a 'TextCursor' does not represent the actual on-screen cursor, so
    #movements do not move the on-screen cursor
    
    #Note 2: when the text is changed, the cursor and selection start/end
    #positions of all cursors are updated accordingly, so the screenCursor
    #stays in place even if characters are inserted at the editCursor
    
    screenCursor = self.textCursor() #For maintaining which region is selected
    editCursor = self.textCursor()   #For inserting the comment marks

    #Use beginEditBlock / endEditBlock to make this one undo/redo operation
    editCursor.beginEditBlock()
    
    try:
        editCursor.setPosition(screenCursor.selectionStart())
        editCursor.movePosition(editCursor.StartOfBlock)
        # < :if selection end is at beginning of the block, don't include that
        #one, except when the selectionStart is same as selectionEnd
        while editCursor.position()<screenCursor.selectionEnd() or \
                editCursor.position()<=screenCursor.selectionStart(): 
            #Create a copy of the editCursor and call the user-supplied function
            editCursorCopy = QtGui.QTextCursor(editCursor)
            function(editCursorCopy)
            
            #Move to the next block
            if not editCursor.block().next().isValid():
                break #We reached the end of the document
            editCursor.movePosition(editCursor.NextBlock)
    finally:
        editCursor.endEditBlock()
.. @+node:ekr.20170108045413.24: *6* doForVisibleBlocks
def doForVisibleBlocks(self, function):
    """ doForVisibleBlocks(function)
    
    Call the given function(cursor) for all blocks that are currently
    visible. This is used by several appearence extensions that
    paint per block.
    
    The supplied cursor will be located at the beginning of each block. This
    cursor may be modified by the function as required
    
    """

    # Start cursor at top line.
    cursor = self.cursorForPosition(QtCore.QPoint(0,0))
    cursor.movePosition(cursor.StartOfBlock)

    while True:            
        # Call the function with a copy of the cursor
        function(QtGui.QTextCursor(cursor))
        
        # Go to the next block (or not if we are done)
        y = self.cursorRect(cursor).bottom() 
        if y > self.height():
            break #Reached end of the repaint area
        if not cursor.block().next().isValid():
            break #Reached end of the text
        cursor.movePosition(cursor.NextBlock)
.. @+node:ekr.20170108045413.25: *6* indentBlock
def indentBlock(self, cursor, amount=1):
    """ indentBlock(cursor, amount=1)
    
    Indent the block given by cursor.
    
    The cursor specified is used to do the indentation; it is positioned
    at the beginning of the first non-whitespace position after completion
    May be overridden to customize indentation.
    
    """
    text = ustr(cursor.block().text())
    leadingWhitespace = text[:len(text)-len(text.lstrip())]
    
    #Select the leading whitespace
    cursor.movePosition(cursor.StartOfBlock)
    cursor.movePosition(cursor.Right,cursor.KeepAnchor,len(leadingWhitespace))
    
    #Compute the new indentation length, expanding any existing tabs
    indent = len(leadingWhitespace.expandtabs(self.indentWidth()))
    if self.indentUsingSpaces():            
        # Determine correction, so we can round to multiples of indentation
        correction = indent % self.indentWidth()
        if correction and amount<0:
            correction = - (self.indentWidth() - correction) # Flip
        # Add the indentation tabs
        indent += (self.indentWidth() * amount) - correction
        cursor.insertText(' '*max(indent,0))
    else:
        # Convert indentation to number of tabs, and add one
        indent = (indent // self.indentWidth()) + amount
        cursor.insertText('\t' * max(indent,0))
.. @+node:ekr.20170108045413.26: *6* dedentBlock
def dedentBlock(self, cursor):
    """ dedentBlock(cursor)
    
    Dedent the block given by cursor.
    
    Calls indentBlock with amount = -1.
    May be overridden to customize indentation.
    
    """
    self.indentBlock(cursor, amount = -1)
.. @+node:ekr.20170108045413.27: *6* indentSelection
def indentSelection(self):
    """ indentSelection()
    
    Called when the current line/selection is to be indented.
    Calls indentLine(cursor) for each line in the selection.
    May be overridden to customize indentation.
    
    See also doForSelectedBlocks and indentBlock.
    
    """
    self.doForSelectedBlocks(self.indentBlock)
.. @+node:ekr.20170108045413.28: *6* dedentSelection
def dedentSelection(self):
    """ dedentSelection()
    
    Called when the current line/selection is to be dedented.
    Calls dedentLine(cursor) for each line in the selection.
    May be overridden to customize indentation.
    
    See also doForSelectedBlocks and dedentBlock.
    
    """
    self.doForSelectedBlocks(self.dedentBlock)
.. @+node:ekr.20170108045413.29: *6* justifyText
def justifyText(self, linewidth=70):
    """ justifyText(linewidth=70)
    """
    from .textutils import TextReshaper
    
    # Get cursor
    cursor = self.textCursor()
    
    # Make selection include whole lines
    pos1, pos2 = cursor.position(), cursor.anchor()
    pos1, pos2 = min(pos1, pos2), max(pos1, pos2)
    cursor.setPosition(pos1, cursor.MoveAnchor)
    cursor.movePosition(cursor.StartOfBlock, cursor.MoveAnchor)
    cursor.setPosition(pos2, cursor.KeepAnchor)
    cursor.movePosition(cursor.EndOfBlock, cursor.KeepAnchor)
    
    # Use reshaper to create replacement text
    reshaper = TextReshaper(linewidth)
    reshaper.pushText(cursor.selectedText())
    newText = reshaper.popText()
    
    # Update the selection
    #self.setTextCursor(cursor) for testing
    cursor.insertText(newText)
.. @+node:ekr.20170108045413.30: *6* addLeftMargin
def addLeftMargin(self, des, func):
    """ Add a margin to the left. Specify a description for the margin,
    and a function to get that margin. For internal use.
    """
    assert des is not None
    self._leftmargins.append((des, func))
.. @+node:ekr.20170108045413.31: *6* getLeftMargin
def getLeftMargin(self, des=None):
    """ Get the left margin, relative to the given description (which
    should be the same as given to addLeftMargin). If des is omitted 
    or None, the full left margin is returned.
    """
    margin = 0
    for d, func in self._leftmargins:
        if d == des:
            break
        margin += func()
    return margin
.. @+node:ekr.20170108045413.32: *6* updateMargins
def updateMargins(self):
    """ Force the margins to be recalculated and set the viewport 
    accordingly.
    """
    leftmargin = self.getLeftMargin()
    self.setViewportMargins(leftmargin , 0, 0, 0)
.. @+node:ekr.20170108050440.1: *3* COPY ../external/pyzo/misc.py
# -*- coding: utf-8 -*-
<< pyzo copyright >>
<< misc.py imports >>
DEFAULT_OPTION_NAME = '_ce_default_value'
DEFAULT_OPTION_NONE = '_+_just some absurd value_+_'
@others
@language python
@tabwidth -4
.. @+node:ekr.20170108171929.1: *4* << pyzo copyright >>
# Copyright (C) 2016, the Pyzo development team
#
# Pyzo is distributed under the terms of the (new) BSD License.
# The full license can be found in 'license.txt'.
.. @+node:ekr.20170108050440.2: *4* << misc.py imports >>
# import sys
import leo.core.leoGlobals as g
ustr = g.ustr
bstr = bytes if g.isPython3 else str
# from .qt import QtGui, QtCore, QtWidgets
from leo.core.leoQt import QtCore, QtWidgets # QtGui, 
if g.isPython3:
    from queue import Queue, Empty
else:
    from Queue import Queue, Empty
###
# # Set Python version and get some names
# PYTHON_VERSION = sys.version_info[0]
# if PYTHON_VERSION < 3:
    # ustr = unicode
    # bstr = str
    # from Queue import Queue, Empty
# else:
    # ustr = str
    # bstr = bytes
    # from queue import Queue, Empty
.. @+node:ekr.20170108050440.3: *4* ce_option
def ce_option(arg1):
    """ Decorator for properties of the code editor. 
    
    It should be used on the setter function, with its default value
    as an argument. The default value is then  stored on the function
    object. 
    
    At the end of the initialization, the base codeeditor class will 
    check all members and (by using the default-value-attribute as a
    flag) select the ones that are options. These are then set to
    their default values.
    
    Similarly this information is used by the setOptions method to
    know which members are "options".
    
    """
    
    # If the decorator is used without arguments, arg1 is the function
    # being decorated. If arguments are used, arg1 is the argument, and
    # we should return a callable that is then used as a decorator.
    
    # Create decorator function.
    def decorator_fun(f):
        f.__dict__[DEFAULT_OPTION_NAME] = default
        return f
    
    # Handle
    default = DEFAULT_OPTION_NONE
    if hasattr(arg1, '__call__'):
        return decorator_fun(arg1)
    else:
        default = arg1
        return decorator_fun
.. @+node:ekr.20170108050440.4: *4* class _CallbackEventHandler
class _CallbackEventHandler(QtCore.QObject):
    """ Helper class to provide the callLater function. 
    """
    
    @others
.. @+node:ekr.20170108050440.5: *5* __init__
def __init__(self):
    QtCore.QObject.__init__(self)
    self.queue = Queue()
.. @+node:ekr.20170108050440.6: *5* customEvent
def customEvent(self, event):
    while True:
        try:
            callback, args = self.queue.get_nowait()
        except Empty:
            break
        try:
            callback(*args)
        except Exception as why:
            print('callback failed: {}:\n{}'.format(callback, why))
.. @+node:ekr.20170108050440.7: *5* postEventWithCallback
def postEventWithCallback(self, callback, *args):
    self.queue.put((callback, args))
    QtWidgets.qApp.postEvent(self, QtCore.QEvent(QtCore.QEvent.User))
.. @+node:ekr.20170108050440.8: *4* callLater
def callLater(callback, *args):
    """ callLater(callback, *args)
    
    Post a callback to be called in the main thread. 
    
    """
    _callbackEventHandler.postEventWithCallback(callback, *args)
    
# Create callback event handler instance and insert function in Pyzo namespace
_callbackEventHandler = _CallbackEventHandler()   
.. @+node:ekr.20170107220425.1: *3* COPY ../external/pyzo/parsers.py
@first # -*- coding: utf-8 -*-
<< pyzo copyright >>
import leo.core.leoGlobals as g
import sys
from .tokens import Token, TextToken
@others
@language python
@tabwidth -4
.. @+node:ekr.20170108171939.1: *4* << pyzo copyright >>
# Copyright (C) 2016, the Pyzo development team
#
# Pyzo is distributed under the terms of the (new) BSD License.
# The full license can be found in 'license.txt'.
.. @+node:ekr.20170107212231.3: *4* class BlockState
class BlockState(object):
    """ BlockState(state=0, info=None)
    
    The blockstate object should be used by parsers to
    return the block state of the processed line. 
    
    This would typically be the last item to be yielded, but this
    it may also be yielded befor the last yielded token. One can even
    yield multiple of these items, in which case the last one considered
    valid.
    
    """
    isToken = False
    @others
# Base parser class (needs to be defined before importing parser modules)
.. @+node:ekr.20170107212231.4: *5* __init__
def __init__(self, state=0, info=None):
    self._state = int(state)
    self._info = info
.. @+node:ekr.20170107212231.5: *5* state
@property
def state(self):
    """ The integer value representing the block state.
    """
    return self._state
.. @+node:ekr.20170107212231.6: *5* info
@property
def info(self):
    """ Get the information corresponding to the block.
    """
    return self._info
.. @+node:ekr.20170107212231.7: *4* class Parser
class Parser(object):
    """ Base parser class. 
    All parsers should inherit from this class.
    This base class generates a 'TextToken' for each line
    """
    _extensions = []
    _keywords = []
    
    
    @others
    
###
## Import parsers statically
# We could load the parser dynamically from the source files in the 
# directory, but this takes quite some effort to get righ when apps 
# are frozen. This is doable (I do it in Visvis) but it requires the
# user to specify the parser modules by hand when freezing an app.
#
# In summary: it takes a lot of trouble, which can be avoided by just
# listing all parsers here.

# from . import (     python_parser, 
                    # cython_parser,
                    # c_parser,
                                # )
.. @+node:ekr.20170107212231.8: *5* p.parseLine
def parseLine(self, line, previousState=0):
    """ parseLine(line, previousState=0)
    
    The method that should be implemented by the parser. The 
    previousState argument can be used to determine how
    the previous block ended (e.g. for multiline comments). It
    is an integer, the meaning of which is only known to the
    specific parser. 
    
    This method should yield token instances. The last token can
    be a BlockState to specify the previousState for the 
    next block.
    
    """
    g.trace('(Parser)')
    yield TextToken(line,0,len(line))

.. @+node:ekr.20170107212231.9: *5* name
def name(self):
    """ name()
    
    Get the name of the parser.
    
    """
    name = self.__class__.__name__.lower()
    if name.endswith('parser'):
        name = name[:-6]
    return name
.. @+node:ekr.20170107212231.10: *5* __repr__
def __repr__(self):
    """ String representation of the parser. 
    """
    return '<Parser for "%s">' % self.name()
.. @+node:ekr.20170107212231.11: *5* keywords
def keywords(self):
    """ keywords()
    
    Get a list of keywords valid for this parser.
    
    """
    return [k for k in self._keywords]
.. @+node:ekr.20170107212231.12: *5* filenameExtensions
def filenameExtensions(self):
    """ filenameExtensions()
    
    Get a list of filename extensions for which this parser
    is appropriate.
    
    """
    return ['.'+e.lstrip('.').lower() for e in self._extensions]
.. @+node:ekr.20170107212231.13: *5* getStyleElementDescriptions
### def getStyleElementDescriptions(cls):
def getStyleElementDescriptions(self):
    """ getStyleElementDescriptions()
    
    This method returns a list of the StyleElementDescription 
    instances used by this parser. 
    
    """
    descriptions = {}
    for token in self.getUsedTokens():
        descriptions[token.description.key] = token.description
    
    return list(descriptions.values())
.. @+node:ekr.20170107212231.14: *5* getUsedTokens
def getUsedTokens(self):
    """ getUsedTokens()
    
    Get a a list of token instances used by this parser.
    
    """
    # Get module object of the parser
    try:
        mod = sys.modules[self.__module__]
    except KeyError:
        return []
    # Get token classes from module
    tokenClasses = []
    for name in mod.__dict__:
        member = mod.__dict__[name]
        if (isinstance(member, type) and 
            issubclass(member, Token)
        ):
            if member is not Token:
                tokenClasses.append(member) 
    # Return as instances
    return [t() for t in tokenClasses]
.. @+node:ekr.20170107212231.15: *5* _isTodoItem
def _isTodoItem(self, text):
    """ _isTodoItem(text)
    
    Get whether the given text (which should be a comment) represents
    a todo item. Todo items start with "todo", "2do" or "fixme", 
    optionally with a colon at the end.
    
    """
    # Get first word
    word = text.lstrip().split(' ',1)[0].rstrip(':')
    # Test
    if word.lower() in ['todo', '2do', 'fixme']:
        return True
    else:
        return False
.. @+node:ekr.20170107213100.1: *3* COPY ../external/pyzo/python_parser.py
@first # -*- coding: utf-8 -*-
<< pyzo copyright >>
<< python_parser imports >>
<< python_parser keywords >>
@others
# if __name__=='__main__':
    # main()
@language python
@tabwidth -4
.. @+node:ekr.20170108171945.1: *4* << pyzo copyright >>
# Copyright (C) 2016, the Pyzo development team
#
# Pyzo is distributed under the terms of the (new) BSD License.
# The full license can be found in 'license.txt'.
.. @+node:ekr.20170107220823.1: *4* << python_parser imports >>
import leo.core.leoGlobals as g
if 1:
    # pylint: disable=no-member
    ustr = str if g.isPython3 else g.builtins.unicode
    text_type = str if g.isPython3 else g.builtins.unicode
import re
from .parsers import Parser, BlockState
from .tokens import ALPHANUM
# Import tokens in module namespace
from .tokens import (CommentToken, StringToken, 
    UnterminatedStringToken, IdentifierToken, NonIdentifierToken,
    KeywordToken, NumberToken, FunctionNameToken, ClassNameToken,
    TodoCommentToken, OpenParenToken, CloseParenToken)
.. @+node:ekr.20170107213124.2: *4* << python_parser keywords >>
# Source: import keyword; keyword.kwlist (Python 2.6.6)
python2Keywords = set([
    'and', 'as', 'assert', 'break', 'class', 'continue', 
    'def', 'del', 'elif', 'else', 'except', 'exec', 'finally', 'for', 
    'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'not', 'or', 
    'pass', 'print', 'raise', 'return', 'try', 'while', 'with', 'yield'])

# Source: import keyword; keyword.kwlist (Python 3.1.2)
python3Keywords = set([
    'False', 'None', 'True', 'and', 'as', 'assert', 'break', 
    'class', 'continue', 'def', 'del', 'elif', 'else', 'except', 'finally', 
    'for', 'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 
    'nonlocal', 'not', 'or', 'pass', 'raise', 'return', 'try', 'while', 
    'with', 'yield'])

# Merge the two sets to get a general Python keyword list        
pythonKeywords = python2Keywords | python3Keywords
.. @+node:ekr.20170107213124.3: *4* class MultilineStringToken (StringToken)
class MultilineStringToken(StringToken):
    """ Characters representing a multi-line string. """
    defaultStyle = 'fore:#7F0000'
.. @+node:ekr.20170107213124.4: *4* class CellCommentToken (CommentToken)
class CellCommentToken(CommentToken):
    """ Characters representing a cell separator comment: "##". """
    defaultStyle = 'bold:yes, underline:yes'


# This regexp is used to find special stuff, such as comments, numbers and
# strings.
# pylint: disable=anomalous-backslash-in-string
tokenProg = re.compile(
    '#|' +						# Comment or
    '([' + ALPHANUM + '_]+)|' +	# Identifiers/numbers (group 1) or
    '(' +  						# Begin of string group (group 2)
    '([bB]|[uU])?' +			# Possibly bytes or unicode (py2.x)
    '[rR]?' +					# Possibly a raw string
    '("""|\'\'\'|"|\')' +		# String start (triple qoutes first, group 4)
    ')|' +						# End of string group
    '(\(|\[|\{)|' +             # Opening parenthesis (gr 5)
    '(\)|\]|\})'                # Closing parenthesis (gr 6)
    )	


#For a given type of string ( ', " , ''' , """ ),get  the RegExp
#program that matches the end. (^|[^\\]) means: start of the line
#or something that is not \ (since \ is supposed to escape the following
#quote) (\\\\)* means: any number of two slashes \\ since each slash will
#escape the next one
endProgs = {
    "'": re.compile(r"(^|[^\\])(\\\\)*'"),
    '"': re.compile(r'(^|[^\\])(\\\\)*"'),
    "'''": re.compile(r"(^|[^\\])(\\\\)*'''"),
    '"""': re.compile(r'(^|[^\\])(\\\\)*"""')
    }
.. @+node:ekr.20170107213124.5: *4* class PythonParser(Parser)
class PythonParser(Parser):
    """ Parser for Python in general (2.x or 3.x).
    """
    _extensions = ['.py' , '.pyw']
    #The list of keywords is overridden by the Python2/3 specific parsers
    _keywords = pythonKeywords 
    
    
    @others
.. @+node:ekr.20170107213124.6: *5* _identifierState
def _identifierState(self, identifier=None):
    """ Given an identifier returs the identifier state:
    3 means the current identifier can be a function.
    4 means the current identifier can be a class.
    0 otherwise.
    
    This method enables storing the state during the line,
    and helps the Cython parser to reuse the Python parser's code.
    """
    if identifier is None:
        # Explicit get/reset
        try:
            # pylint: disable=access-member-before-definition
            state = self._idsState
        except Exception:
            state = 0
        self._idsState = 0
        return state
    elif identifier == 'def':
        # Set function state
        self._idsState = 3
        return 3
    elif identifier == 'class':
        # Set class state
        self._idsState = 4
        return 4
    else:
        # This one can be func or class, next one can't
        state = self._idsState
        self._idsState = 0
        return state
.. @+node:ekr.20170107213124.7: *5* py.parseLine
n_parse = 0
n_tokens = 0

def parseLine(self, line, previousState=0):
    """ parseLine(line, previousState=0)
    
    Parse a line of Python code, yielding tokens.
    previousstate is the state of the previous block, and is used
    to handle line continuation and multiline strings.
    
    """
    self.n_parse += 1
    line = text_type(line)
    # Init
    pos = 0 # Position following the previous match
    # identifierState and previousstate values:
    # 0: nothing special
    # 1: multiline comment single qoutes
    # 2: multiline comment double quotes
    # 3: a def keyword
    # 4: a class keyword
    
    #Handle line continuation after def or class
    #identifierState is 3 or 4 if the previous identifier was 3 or 4
    if previousState == 3 or previousState == 4: 
        self._identifierState({3:'def',4:'class'}[previousState])
    else:
        self._identifierState(None)
    if previousState in [1,2]:
        token = MultilineStringToken(line, 0, 0)
        token._style = ['', "'''", '"""'][previousState]
        tokens = self._findEndOfString(line, token)
        # Process tokens
        for token in tokens:
            yield token
            if isinstance(token, BlockState):
                return 
        pos = token.end
    # Enter the main loop that iterates over the tokens and skips strings
    while True:
        # Get next tokens
        tokens = self._findNextToken(line, pos)
        if not tokens:
            return
        elif isinstance(tokens[-1], StringToken):
            moreTokens = self._findEndOfString(line, tokens[-1])
            tokens = tokens[:-1] + moreTokens
        # Process tokens
        for token in tokens:
            yield token
            if isinstance(token, BlockState):
                return 
        pos = token.end
.. @+node:ekr.20170107213124.8: *5* _findEndOfString
def _findEndOfString(self, line, token):
    """ _findEndOfString(line, token)
    
    Find the end of a string. Returns (token, endToken). The first 
    is the given token or a replacement (UnterminatedStringToken).
    The latter is None, or the BlockState. If given, the line is
    finished.
    
    """
    
    # Set state
    self._identifierState(None)
    
    # Find the matching end in the rest of the line
    # Do not use the start parameter of search, since ^ does not work then
    style = token._style
    endMatch = endProgs[style].search(line[token.end:])
    
    if endMatch:
        # The string does end on this line
        tokenArgs = line, token.start, token.end + endMatch.end()
        if style in ['"""', "'''"]:
            token = MultilineStringToken(*tokenArgs)
        else:
            token.end = token.end + endMatch.end()
        return [token]
    else:
        # The string does not end on this line
        tokenArgs = line, token.start, token.end + len(line)
        if style == "'''":
            return [MultilineStringToken(*tokenArgs), BlockState(1)]
        elif style == '"""':
            return [MultilineStringToken(*tokenArgs), BlockState(2)]
        else:
            return [UnterminatedStringToken(*tokenArgs)]
.. @+node:ekr.20170107213124.9: *5* _findNextToken
def _findNextToken(self, line, pos):
    """ _findNextToken(line, pos):
    
    Returns a token or None if no new tokens can be found.
    
    """
    
    # Init tokens, if pos too large, were done
    if pos > len(line):
        return None
    tokens = []
    
    # Find the start of the next string or comment
    match = tokenProg.search(line, pos)
    
    # Process the Non-Identifier between pos and match.start() 
    # or end of line
    nonIdentifierEnd = match.start() if match else len(line)
    
    # Return the Non-Identifier token if non-null
    # todo: here it goes wrong (allow returning more than one token?)
    token = NonIdentifierToken(line,pos,nonIdentifierEnd)
    strippedNonIdentifier = ustr(token).strip()
    if token:
        tokens.append(token)
    
    # Do checks for line continuation and identifierState
    # Is the last non-whitespace a line-continuation character?
    if strippedNonIdentifier.endswith('\\'):
        lineContinuation = True
        # If there are non-whitespace characters after def or class,
        # cancel the identifierState
        if strippedNonIdentifier != '\\':
            self._identifierState(None)
    else:
        lineContinuation = False
        # If there are non-whitespace characters after def or class,
        # cancel the identifierState
        if strippedNonIdentifier != '':
            self._identifierState(None)
    
    # If no match, we are done processing the line
    if not match:
        if lineContinuation:
            tokens.append( BlockState(self._identifierState()) )
        return tokens
    
    # The rest is to establish what identifier we are dealing with
    
    # Comment
    if match.group() == '#':
        matchStart = match.start()
        if not line[:matchStart].strip() and (
               line[matchStart:].startswith('##') or 
               line[matchStart:].startswith('#%%') or 
               line[matchStart:].startswith('# %%')):
            tokens.append( CellCommentToken(line,matchStart,len(line)) )
        elif self._isTodoItem(line[matchStart+1:]):
            tokens.append( TodoCommentToken(line,matchStart,len(line)) )
        else:
            tokens.append( CommentToken(line,matchStart,len(line)) )
        if lineContinuation:
            tokens.append( BlockState(self._identifierState()) )
        return tokens
    
    # If there are non-whitespace characters after def or class,
    # cancel the identifierState (this time, also if there is just a \
    # since apparently it was not on the end of a line)
    if strippedNonIdentifier != '':
        self._identifierState(None)
    
    # Identifier ("a word or number") Find out whether it is a key word
    if match.group(1) is not None:
        identifier = match.group(1)
        tokenArgs = line, match.start(), match.end()
        
        # Set identifier state 
        identifierState = self._identifierState(identifier)
        
        if identifier in self._keywords:
            tokens.append( KeywordToken(*tokenArgs) )
        elif identifier[0] in '0123456789':
            self._identifierState(None)
            tokens.append( NumberToken(*tokenArgs) )
        else:
            if (identifierState==3 and
                    line[match.end():].lstrip().startswith('(') ):
                tokens.append( FunctionNameToken(*tokenArgs) )
            elif identifierState==4:
                tokens.append( ClassNameToken(*tokenArgs) )
            else:
                tokens.append( IdentifierToken(*tokenArgs) )
    
    elif match.group(2) is not None :
        # We have matched a string-start
        # Find the string style ( ' or " or ''' or """)
        token = StringToken(line, match.start(), match.end())
        token._style = match.group(4) # The style is in match group 4
        tokens.append( token )
    elif match.group(5) is not None :
        token = OpenParenToken(line, match.start(), match.end())
        token._style = match.group(5)
        tokens.append(token)
    elif match.group(6) is not None :
        token = CloseParenToken(line, match.start(), match.end())
        token._style = match.group(6) 
        tokens.append(token)
    # Done
    return tokens
.. @+node:ekr.20170107213124.10: *4* class Python2Parser(Parser)
class Python2Parser(PythonParser):
    """ Parser for Python 2.x code.
    """
     # The application should choose whether to set the Py 2 specific parser
    _extensions = []
    _keywords = python2Keywords
.. @+node:ekr.20170107213124.11: *4* class Python3Parser(Parser)
class Python3Parser(PythonParser):
    """ Parser for Python 3.x code.
    """
    # The application should choose whether to set the Py 3 specific parser
    _extensions = []
    _keywords = python3Keywords
.. @+node:ekr.20170107222650.1: *4* main
def main():
    pass # tokenizeLine is undefined.
    # print(list(tokenizeLine('this is "String" #Comment')))
    # print(list(tokenizeLine('this is "String\' #Comment')))
    # print(list(tokenizeLine('this is "String\' #Commen"t')))
    # print(list(tokenizeLine(r'this "test\""')))
    # import random
    # stimulus=''
    # expect=[]
    # for i in range(10):
        # #Create a string with lots of ' and "
        # s=''.join("'\"\\ab#"[random.randint(0,5)] for i in range(10)  )
        # stimulus+=repr(s)
        # expect.append('S:'+repr(s))
        # stimulus+='test'
        # expect.append('I:test')
    # result=list(tokenizeLine(stimulus))
    # print (stimulus)
    # print (expect)
    # print (result)
    # assert repr(result) == repr(expect)
.. @+node:ekr.20170107214315.1: *3* COPY ../external/pyzo/style.py
@first # -*- coding: utf-8 -*-
<< pyzo copyright >>
from leo.core.leoQt import QtCore, QtGui
Qt = QtCore.Qt
@others
@language python
@tabwidth -4
.. @+node:ekr.20170108171955.1: *4* << pyzo copyright >>
# Copyright (C) 2016, the Pyzo development team
#
# Pyzo is distributed under the terms of the (new) BSD License.
# The full license can be found in 'license.txt'.
.. @+node:ekr.20170107214321.3: *4* class StyleElementDescription
class StyleElementDescription:
    """ StyleElementDescription(name, defaultFormat, description)
    
    Describes a style element by its name, default format, and description.
    
    A style description is a simple placeholder for something
    that can be styled.
    
    """
    
    @others
.. @+node:ekr.20170107214321.4: *5* sf.__init__
def __init__(self, name, description, defaultFormat):
    self._name = name
    self._description = description
    self._defaultFormat = StyleFormat(defaultFormat)
.. @+node:ekr.20170107214321.5: *5* __repr__
def __repr__(self):
    return '<"%s": "%s">' % (self.name, self.defaultFormat)
.. @+node:ekr.20170107214321.6: *5* name
@property
def name(self):
    return self._name
.. @+node:ekr.20170107214321.7: *5* key
@property
def key(self):
    return self._name.replace(' ', '').lower()
.. @+node:ekr.20170107214321.8: *5* description
@property
def description(self):
    return self._description
.. @+node:ekr.20170107214321.9: *5* defaultFormat
@property
def defaultFormat(self):
    return self._defaultFormat
.. @+node:ekr.20170107214321.10: *4* class StyleFormat
class StyleFormat:
    """ StyleFormat(format='')
    
    Represents the style format for a specific style element.
    A "style" is a dictionary that maps names (of style elements) 
    to StyleFormat instances.
    
    The given format can be a string or another StyleFormat instance.
    Style formats can be combined using their update() method. 
    
    A style format consists of multiple parts, where each "part" consists
    of a key and a value. The keys can be anything, depending
    on what kind of thing is being styled. The value can be obtained using
    the index operator (e.g. styleFomat['fore'])
    
    For a few special keys, properties are defined that return the Qt object
    corresponding to the value. These values are also buffered to enable
    fast access. These keys are:
      * fore: (QColor) the foreground color
      * back: (QColor) the background color
      * bold: (bool) whether the text should be bold
      * italic: (bool) whether the text should be in italic
      * underline: (int) whether an underline should be used (and which one)
      * linestyle: (int) what line style to use (e.g. for indent guides)
      * textCharFOrmat: (QTextCharFormat) for the syntax styles
    
    The format neglects spaces and case. Parts are separated by commas 
    or semicolons. If only a key is given it's value is interpreted
    as 'yes'. If only a color is given, its key is interpreted as 'fore' 
    and back. Colors should be given using the '#' hex formatting.
    
    An example format string: 'fore:#334, bold, underline:dotLine'
    
    By calling str(styleFormatInstance) the string representing of the 
    format can be obtained. By iterating over the instance, a series 
    of key-value pairs is obtained.
    
    """
    
    @others
.. @+node:ekr.20170107214321.11: *5* sf.__init__
def __init__(self, format=''):
    self._parts = {}
    self.update(format)
.. @+node:ekr.20170107214321.12: *5* sf._resetProperties
def _resetProperties(self):
    self._fore = None
    self._back = None
    self._bold = None
    self._italic = None
    self._underline = None
    self._linestyle = None
    self._textCharFormat = None
.. @+node:ekr.20170107214321.13: *5* sf.__str__
def __str__(self):
    """ Get a (cleaned up) string representation of this style format. 
    """
    parts = []
    for key in self._parts:
        parts.append('%s:%s' % (key, self._parts[key]))
    return ', '.join(parts)
.. @+node:ekr.20170107214321.14: *5* sf.__repr__
def __repr__(self):
    return '<StyleFormat "%s">' % str(self)
.. @+node:ekr.20170107214321.15: *5* sf.__getitem__
def __getitem__(self, key):
    try:
        return self._parts[key]
    except KeyError:
        raise KeyError('Invalid part key for style format.')
.. @+node:ekr.20170107214321.16: *5* sf.__iter__
def __iter__(self):
    """ Yields a series of tuples (key, val).
    """
    parts = []
    for key in self._parts:
        parts.append( (key, self._parts[key]) )
    return parts.__iter__()
.. @+node:ekr.20170107214321.17: *5* sf.update
def update(self, format):
    """ update(format)
    
    Update this style format with the given format.
    
    """
    # Reset buffered values
    self._resetProperties()
    # Make a string, so we update the format with the given one
    if isinstance(format, StyleFormat):
        format = str(format)
    # Split on ',' and ',', ignore spaces
    styleParts = [p for p in
        format.replace('=',':').replace(';',',').split(',')]
    for stylePart in styleParts:
        # Make sure it consists of identifier and value pair
        # e.g. fore:#xxx, bold:yes, underline:no
        if not ':' in stylePart:
            if stylePart.startswith('#'):
                stylePart = 'foreandback:' + stylePart
            else:
                stylePart += ':yes'
        # Get key value and strip and make lowecase
        key, _, val = [i.strip().lower() for i in stylePart.partition(':')]
        # Store in parts
        if key == 'foreandback':
            self._parts['fore'] = val
            self._parts['back'] = val
        elif key:
            self._parts[key] = val
.. @+node:ekr.20170108171245.1: *5* properties & helper
.. @+node:ekr.20170107214321.18: *6* _getValueSafe
def _getValueSafe(self, key):
    try:
        return self._parts[key]
    except KeyError:
        return 'no'
.. @+node:ekr.20170107214321.20: *6* back
@property
def back(self):
    if self._back is None:
        self._back = QtGui.QColor(self._parts['back'])
    return self._back
.. @+node:ekr.20170107214321.21: *6* bold
@property
def bold(self):
    if self._bold is None:
        if self._getValueSafe('bold') in ['yes', 'true']:
            self._bold = True
        else:
            self._bold = False
    return self._bold
.. @+node:ekr.20170107214321.19: *6* fore
@property
def fore(self):
    if self._fore is None:
        self._fore = QtGui.QColor(self._parts['fore'])
    return self._fore
.. @+node:ekr.20170107214321.22: *6* italic
@property
def italic(self):
    if self._italic is None:
        if self._getValueSafe('italic') in ['yes', 'true']:
            self._italic = True
        else:
            self._italic = False
    return self._italic
.. @+node:ekr.20170107214321.24: *6* linestyle
@property
def linestyle(self):
    if self._linestyle is None:
        val = self._getValueSafe('linestyle')
        if val in ['yes', 'true']:
            self._linestyle = Qt.SolidLine
        elif val in ['dotted', 'dot', 'dots', 'dotline']: 
            self._linestyle = Qt.DotLine
        elif val in ['dashed', 'dash', 'dashes', 'dashline']: 
            self._linestyle = Qt.DashLine
        else:
            self._linestyle = Qt.SolidLine # default to solid
    return self._linestyle
.. @+node:ekr.20170107214321.25: *6* textCharFormat
@property
def textCharFormat(self):
    if self._textCharFormat is None:
        self._textCharFormat = tcf = QtGui.QTextCharFormat()
        self._textCharFormat.setForeground(self.fore)
        self._textCharFormat.setUnderlineStyle(self.underline)
        if self.bold:
            self._textCharFormat.setFontWeight(QtGui.QFont.Bold)
        if self.italic:
            self._textCharFormat.setFontItalic(True)
    return self._textCharFormat
.. @+node:ekr.20170107214321.23: *6* underline
@property
def underline(self):
    if self._underline is None:
        val = self._getValueSafe('underline')
        if val in ['yes', 'true']:
            self._underline = QtGui.QTextCharFormat.SingleUnderline
        elif val in ['dotted', 'dots', 'dotline']: 
            self._underline = QtGui.QTextCharFormat.DotLine
        elif val in ['wave']: 
            self._underline = QtGui.QTextCharFormat.WaveUnderline
        else:
            self._underline = QtGui.QTextCharFormat.NoUnderline
    return self._underline
.. @+node:ekr.20170107212657.1: *3* COPY ../external/pyzo/tokens.py
@first # -*- coding: utf-8 -*-
<< pyzo copyright >>
<< tokens imports >>
ALPHANUM = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789'
@others
@language python
@tabwidth -4
.. @+node:ekr.20170108171959.1: *4* << pyzo copyright >>
# Copyright (C) 2016, the Pyzo development team
#
# Pyzo is distributed under the terms of the (new) BSD License.
# The full license can be found in 'license.txt'.
.. @+node:ekr.20170107220236.1: *4*   << tokens imports >>
import leo.core.leoGlobals as g
if 1:
    # pylint: disable=no-member
    ustr = str if g.isPython3 else g.builtins.unicode
from .style import StyleFormat, StyleElementDescription
.. @+node:ekr.20170107212709.3: *4*  class Token
class Token(object):
    """ Token(line, start, end)
    
    Base token class.
    
    A token is a group of characters representing "something".
    What is represented, is specified by the subclass.
    
    Each token class should have a docstring describing the meaning
    of the characters it is applied to.
    
    """ 
    defaultStyle = 'fore:#000, bold:no, underline:no, italic:no'
    isToken = True # For the BlockState object, which is also returned by the parsers, this is False
    @others
.. @+node:ekr.20170107212709.4: *5* __init__
def __init__(self, line='', start=0, end=0):
    self.line = ustr(line)
    self.start = start
    self.end = end
    self._name = self._getName()
.. @+node:ekr.20170107212709.5: *5* __str__
def __str__(self):  # on 2.x we use __unicode__
    return self.line[self.start:self.end]
.. @+node:ekr.20170107212709.6: *5* __unicode__
def __unicode__(self):  # for py 2.x
    return self.line[self.start:self.end]
.. @+node:ekr.20170107212709.7: *5* __repr__
def __repr__(self):
    return repr('%s:%s' % (self.name, self))
.. @+node:ekr.20170107212709.8: *5* __len__
def __len__(self):
    # Defining a length also gives a Token a boolean value: True if there
    # are any characters (len!=0) and False if there are none
    return self.end - self.start
.. @+node:ekr.20170107212709.9: *5* _getName
def _getName(self):
    """ Get the name of this token. """
    nameParts = ['Syntax']
    if '_parser' in self.__module__:
        language = self.__module__.split('_')[0]
        language = language.split('.')[-1]
        nameParts.append( language[0].upper() + language[1:] )
    nameParts.append( self.__class__.__name__[:-5].lower() )
    return '.'.join(nameParts)
.. @+node:ekr.20170107212709.10: *5* tok.getDefaultStyleFormat
def getDefaultStyleFormat(self):
    elements = []
    def collect(cls):
        if hasattr(cls, 'defaultStyle'):
            elements.append(cls.defaultStyle)
            for c in cls.__bases__:
                collect(c)
    collect(self.__class__)
    se = StyleFormat()
    for e in reversed(elements):
        se.update(e)
    return se
.. @+node:ekr.20170107212709.11: *5* name
@property
def name(self):
    """ The name of this token. Used to identify it and attach a style.
    """
    return self._name
.. @+node:ekr.20170107212709.12: *5* description
@property
def description(self):
    """ description()
    
    Returns a StyleElementDescription instance that describes the
    style element that this token represents.
    
    """
    format = self.getDefaultStyleFormat()
    des = 'syntax: ' + self.__doc__
    return StyleElementDescription(self.name, des, str(format))
.. @+node:ekr.20170107212709.13: *4* class CommentToken & subclasses
class CommentToken(Token):
    """ Characters representing a comment in the code. """
    defaultStyle = 'fore:#007F00'
.. @+node:ekr.20170107212709.14: *5* class TodoCommentToken
class TodoCommentToken(CommentToken):
    """ Characters representing a comment in the code. """
    defaultStyle = 'fore:#E00,italic'
.. @+node:ekr.20170107212709.24: *4* class ParenthesisToken & subclasses
class ParenthesisToken(Token) :
    """ Parenthesis (and square and curly brackets). """
    defaultStyle = ''
.. @+node:ekr.20170107212709.25: *5* class OpenParenToken
class OpenParenToken(ParenthesisToken) :
    """ Opening parenthesis (and square and curly brackets). """
    defaultStyle = ''
.. @+node:ekr.20170107212709.26: *5* class CloseParenToken
class CloseParenToken(ParenthesisToken) :
    """ Closing parenthesis (and square and curly brackets). """
    defaultStyle = ''
.. @+node:ekr.20170107212709.15: *4* class StringToken & subclasses
class StringToken(Token):
    """ Characters representing a textual string in the code. """
    defaultStyle = 'fore:#7F007F'
.. @+node:ekr.20170107212709.16: *5* class UnterminatedStringToken
class UnterminatedStringToken(StringToken):
    """ Characters belonging to an unterminated string. """
    defaultStyle = 'underline:dotted'

# todo: request from user: whitespace token
.. @+node:ekr.20170107212709.17: *4* class TextToken & subclasses
class TextToken(Token):
    """ Anything that is not a string or comment. """ 
    defaultStyle = 'fore:#000'
.. @+node:ekr.20170107212709.18: *5* class IdentifierToken & subclasses
class IdentifierToken(TextToken):
    """ Characters representing normal text (i.e. words). """ 
    defaultStyle = ''
.. @+node:ekr.20170107212709.23: *6* class ClassNameToken
class ClassNameToken(IdentifierToken):
    """ Characters represening the name of a class. """
    defaultStyle = 'fore:#0000FF, bold:yes'
.. @+node:ekr.20170107212709.22: *6* class FunctionNameToken
class FunctionNameToken(IdentifierToken):
    """ Characters represening the name of a function. """
    defaultStyle = 'fore:#007F7F, bold:yes'
.. @+node:ekr.20170107212709.20: *6* class KeywordToken
class KeywordToken(IdentifierToken):
    """ A keyword is a word with a special meaning to the language. """
    defaultStyle = 'fore:#00007F, bold:yes'
.. @+node:ekr.20170107212709.19: *6* class NonIdentifierToken
class NonIdentifierToken(TextToken):
    """ Not a word (operators, whitespace, etc.). """
    defaultStyle = ''
.. @+node:ekr.20170107212709.21: *6* class NumberToken
class NumberToken(IdentifierToken):
    """ Characters represening a number. """
    defaultStyle = 'fore:#007F7F'
.. @+node:ekr.20170302124438.1: ** Recent code
@language python
.. @+node:ekr.20170203120944.1: *3* jedit.match_image
image_url = re.compile(r'^\s*<\s*img\s+.*src=\"(.*)\".*>\s*$')

def match_image(self, s, i):
    '''Matcher for <img...>'''
    if i == 0:
        m = self.image_url.match(s)
        if m:
            self.image_src = src = m.group(1)
            j = len(src)
            g.trace('Found. length', j, repr(src))
            # body = self.c.frame.body
            # widget, wrapper = body.widget, body.wrapper
            # s = wrapper.getAllText()
            # wrapper.delete(0, j)
            # widget.insertHtml(src)
            return j
        else:
            return 0
    else:
        return 0
.. @+node:ekr.20170204164635.1: *3* class ImageManager
class ImageManager:
    '''A global class to manage persistent images in the body pane.'''
    
    def __init__(self):
        '''ImageManager.__init__'''
        self.cursors = []
        self.images = []
        self.names = []
        self.name_index = 0
        
    @others
        
.. @+node:ekr.20170204164826.1: *4* image.deselect
def deselect (self, c):
    '''Save image data when unselecting c's body pane.'''
    # widget = c.frame.body.widget
    g.trace()
.. @+node:ekr.20170204164839.1: *4* image.select
def select (self, c):
    '''Prepare all images in c's body pane.'''
    # widget = c.frame.body.widget
    g.trace()
.. @+node:ekr.20170303113048.1: *3* c.openLeoThemes
@cmd('open-themes-leo')
def openLeoThemes(self, event=None):
    '''Open themes.leo in a new Leo window.'''
    c = self
    fn = g.os_path_finalize_join(g.app.loadDir, '..', 'config', 'themes.leo')
    if g.os_path_exists(fn):
        return g.openWithFileName(fn, old_c=c)
    else:
        g.es('file not found: %s' % fn)
        return None
.. @-all
.. @@nosearch
.. @-leo
