From pianohacker at gmail.com Fri Jan 5 13:48:13 2007 From: pianohacker at gmail.com (Jesse Weaver) Date: Fri, 5 Jan 2007 16:48:13 -0700 Subject: [Sdl-developers] ABNF Grammar Message-ID: An early version of the grammaris up, supporting everything but multiline comments. Any ideas on how we could add these? -- Jesse ----------- There is a theory which states that if anybody ever discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable. There is another theory which states that this has already happened. - Douglas Adams, author of Hitchhiker's Guide to the Galaxy -------------- next part -------------- An HTML attachment was scrubbed... URL: From dan at ikayzo.com Fri Jan 5 15:34:13 2007 From: dan at ikayzo.com (Daniel Leuck) Date: Fri, 5 Jan 2007 15:34:13 -1000 Subject: [Sdl-developers] ABNF Grammar In-Reply-To: Message-ID: Hi Jesse, This is a great contribution! I added the lowercase versions of the letter suffixes for long, float, double, and Big Decimal literals. > Any ideas on how we could add these? Not off hand. I haven't worked with ABNF enough. Eric or Pat, do you know? Dan _____ From: sdl-developers-bounces at ikayzo.org [mailto:sdl-developers-bounces at ikayzo.org] On Behalf Of Jesse Weaver Sent: Friday, January 05, 2007 1:48 PM To: SDL_Developers Subject: [Sdl-developers] ABNF Grammar An early version of the grammar is up, supporting everything but multiline comments. Any ideas on how we could add these? -- Jesse ----------- There is a theory which states that if anybody ever discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable. There is another theory which states that this has already happened. - Douglas Adams, author of Hitchhiker's Guide to the Galaxy -------------- next part -------------- An HTML attachment was scrubbed... URL: From pianohacker at gmail.com Sat Jan 6 12:11:42 2007 From: pianohacker at gmail.com (Jesse Weaver) Date: Sat, 6 Jan 2007 15:11:42 -0700 Subject: [Sdl-developers] ABNF Grammar In-Reply-To: <459efc9b.070b7084.454a.ffffb076SMTPIN_ADDED@mx.google.com> References: <459efc9b.070b7084.454a.ffffb076SMTPIN_ADDED@mx.google.com> Message-ID: Actually, how is support for multiline comments implemented in the Java and .NET versions? -- Jesse ----------- There is a theory which states that if anybody ever discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable. There is another theory which states that this has already happened. - Douglas Adams, author of Hitchhiker's Guide to the Galaxy -------------- next part -------------- An HTML attachment was scrubbed... URL: From dan at ikayzo.com Sat Jan 6 12:28:05 2007 From: dan at ikayzo.com (Daniel Leuck) Date: Sat, 6 Jan 2007 12:28:05 -1000 Subject: [Sdl-developers] ABNF Grammar In-Reply-To: Message-ID: Hi Jesse, They both work exactly as specified (the same as multiline comments in C or Java.) Dan _____ From: sdl-developers-bounces at ikayzo.org [mailto:sdl-developers-bounces at ikayzo.org] On Behalf Of Jesse Weaver Sent: Saturday, January 06, 2007 12:12 PM To: sdl-developers at ikayzo.org:Information relating to SDL development Subject: Re: [Sdl-developers] ABNF Grammar Actually, how is support for multiline comments implemented in the Java and .NET versions? -- Jesse ----------- There is a theory which states that if anybody ever discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable. There is another theory which states that this has already happened. - Douglas Adams, author of Hitchhiker's Guide to the Galaxy -------------- next part -------------- An HTML attachment was scrubbed... URL: From dan at ikayzo.com Sat Jan 6 16:48:58 2007 From: dan at ikayzo.com (Daniel Leuck) Date: Sat, 6 Jan 2007 16:48:58 -1000 Subject: [Sdl-developers] ABNF Grammar Message-ID: Hi Jesse, Sorry - I misread your question the first time. Tokenization occurs in a method called getLineTokens() in the Parser class. Inside this method multiline comments are handled by calling another method called handleSlashComment(). It consumes characters (including line terminators) until it finds "*/". I will add more comments to the parser class in the near future. Dan _____ From: Daniel Leuck [mailto:dan at ikayzo.com] Sent: Saturday, January 06, 2007 12:28 PM To: 'sdl-developers at ikayzo.org:Information relating to SDL development' Subject: RE: [Sdl-developers] ABNF Grammar Hi Jesse, They both work exactly as specified (the same as multiline comments in C or Java.) Dan _____ From: sdl-developers-bounces at ikayzo.org [mailto:sdl-developers-bounces at ikayzo.org] On Behalf Of Jesse Weaver Sent: Saturday, January 06, 2007 12:12 PM To: sdl-developers at ikayzo.org:Information relating to SDL development Subject: Re: [Sdl-developers] ABNF Grammar Actually, how is support for multiline comments implemented in the Java and .NET versions? -- Jesse ----------- There is a theory which states that if anybody ever discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable. There is another theory which states that this has already happened. - Douglas Adams, author of Hitchhiker's Guide to the Galaxy -------------- next part -------------- An HTML attachment was scrubbed... URL: From subcallnop at yahoo.com Sun Jan 7 21:58:46 2007 From: subcallnop at yahoo.com (E M) Date: Sun, 7 Jan 2007 23:58:46 -0800 (PST) Subject: [Sdl-developers] ABNF Grammar In-Reply-To: Message-ID: <580201.19696.qm@web50404.mail.yahoo.com> Jesse, I remember that the Lex and Yacc Oreilly book (which I unfortunately gave away) had an example of lexing multiline comments out of the token stream. It was done using "start conditions" which basically define a sublexer to parse the comment out separate from the language definition. I know that no one is probably using Flex anymore, but if you are writing a lexer, there might be a hint in the Flex manual. There is a specific example of parsing C comments with Flex in section 0.11 and 0.18. Also, I took a look at the official C grammer, and the official Java grammar, and noticed that both do NOT contain a description of the comment lexing. So maybe it does not need a description in the language definition of SDL. Eric Jesse Weaver wrote: An early version of the grammar is up, supporting everything but multiline comments. Any ideas on how we could add these? -- Jesse ----------- There is a theory which states that if anybody ever discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable. There is another theory which states that this has already happened. - Douglas Adams, author of Hitchhiker's Guide to the Galaxy _______________________________________________ Sdl-developers mailing list Sdl-developers at ikayzo.org http://ml.ikayzo.org/mailman/listinfo/sdl-developers "Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety." -- B. Franklin -------------- next part -------------- An HTML attachment was scrubbed... URL: From dan at ikayzo.com Sun Jan 7 23:15:47 2007 From: dan at ikayzo.com (Daniel Leuck) Date: Sun, 7 Jan 2007 23:15:47 -1000 Subject: [Sdl-developers] ABNF Grammar In-Reply-To: <580201.19696.qm@web50404.mail.yahoo.com> Message-ID: Hi Eric, Thank you for the feedback. Jesse - perhaps your comment above the definition noting that multi-line comments are not covered is sufficient. Dan _____ From: sdl-developers-bounces at ikayzo.org [mailto:sdl-developers-bounces at ikayzo.org] On Behalf Of E M Sent: Sunday, January 07, 2007 9:59 PM To: sdl-developers at ikayzo.org:Information relating to SDL development Subject: Re: [Sdl-developers] ABNF Grammar Jesse, I remember that the Lex and Yacc Oreilly book (which I unfortunately gave away) had an example of lexing multiline comments out of the token stream. It was done using "start conditions" which basically define a sublexer to parse the comment out separate from the language definition. I know that no one is probably using Flex anymore, but if you are writing a lexer, there might be a hint in the Flex manual. There is a specific example of parsing C comments with Flex in section 0.11 and 0.18. Also, I took a look at the official C grammer, and the official Java grammar, and noticed that both do NOT contain a description of the comment lexing. So maybe it does not need a description in the language definition of SDL. Eric Jesse Weaver wrote: An early version of the grammar is up, supporting everything but multiline comments. Any ideas on how we could add these? -- Jesse ----------- There is a theory which states that if anybody ever discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable. There is another theory which states that this has already happened. - Douglas Adams, author of Hitchhiker's Guide to the Galaxy _______________________________________________ Sdl-developers mailing list Sdl-developers at ikayzo.org http://ml.ikayzo.org/mailman/listinfo/sdl-developers "Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety." -- B. Franklin -------------- next part -------------- An HTML attachment was scrubbed... URL: From pianohacker at gmail.com Mon Jan 8 08:06:27 2007 From: pianohacker at gmail.com (Jesse Weaver) Date: Mon, 8 Jan 2007 10:06:27 -0800 Subject: [Sdl-developers] ABNF Grammar In-Reply-To: <45a20bc9.0ad11918.4297.ffff8e41SMTPIN_ADDED@mx.google.com> References: <580201.19696.qm@web50404.mail.yahoo.com> <45a20bc9.0ad11918.4297.ffff8e41SMTPIN_ADDED@mx.google.com> Message-ID: I guess it'll have to be. Thanks for the advice, all. On 1/8/07, Daniel Leuck wrote: > > Hi Eric, > > > > Thank you for the feedback. Jesse - perhaps your comment above the > definition noting that multi-line comments are not covered is sufficient. > > > _______________________________________________ > Sdl-developers mailing list > Sdl-developers at ikayzo.org > http://ml.ikayzo.org/mailman/listinfo/sdl-developers > > > -- Jesse ----------- There is a theory which states that if anybody ever discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable. There is another theory which states that this has already happened. - Douglas Adams, author of Hitchhiker's Guide to the Galaxy -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdb at getsu.com Mon Jan 8 09:09:08 2007 From: jdb at getsu.com (J. David Beutel) Date: Mon, 08 Jan 2007 09:09:08 -1000 Subject: [Sdl-developers] ABNF Grammar In-Reply-To: References: <580201.19696.qm@web50404.mail.yahoo.com> <45a20bc9.0ad11918.4297.ffff8e41SMTPIN_ADDED@mx.google.com> Message-ID: <45A296D4.5010702@getsu.com> Java defines two separate grammars: lexical and syntactic. The comments are in the lexical grammar, along with Unicode escapes and whitespace. It's applied separately, before the syntactic grammar. http://java.sun.com/docs/books/jls/third_edition/html/grammars.html#151185 I guess this follows the tradition of Lex and Yacc, and many languages do it the same way. I would have assumed that the distinction was for performance reasons, but since performance is so much better nowadays, I guess it also makes it easier to understand. So I wouldn't worry about unifying it for SDL. Cheers, 11011011 "Jesse Weaver" wrote on Monday, January 08, 2007 08:06:27: > I guess it'll have to be. Thanks for the advice, all. > > On 1/8/07, *Daniel Leuck* > wrote: > > Hi Eric, > > > > Thank you for the feedback. Jesse - perhaps your comment above > the definition noting that multi-line comments are not covered is > sufficient. > > > > _______________________________________________ > Sdl-developers mailing list > Sdl-developers at ikayzo.org > http://ml.ikayzo.org/mailman/listinfo/sdl-developers > > > > > -- > Jesse > > ----------- > There is a theory which states that if anybody ever discovers exactly > what the Universe is for and why it is here, it will instantly > disappear and be replaced by something even more bizarre and > inexplicable. There is another theory which states that this has > already happened. - Douglas Adams, author of Hitchhiker's Guide to the > Galaxy > ------------------------------------------------------------------------ > > _______________________________________________ > Sdl-developers mailing list > Sdl-developers at ikayzo.org > http://ml.ikayzo.org/mailman/listinfo/sdl-developers > From eric at sulawesi.net Mon Jan 8 14:19:04 2007 From: eric at sulawesi.net (E M) Date: Mon, 8 Jan 2007 16:19:04 -0800 (PST) Subject: [Sdl-developers] ABNF Grammar In-Reply-To: <45A296D4.5010702@getsu.com> Message-ID: <251844.35644.qm@web50409.mail.yahoo.com> David, You are right. I was looking at the wrong Java grammar. Also, the C grammar predates what we would consider reasonably fast machines by a few decades. At the time it would have made more sense to "optimize" the comment handling out of the grammar, and use lexer tricks. This might not be an option for SDL's grammar, especially if the Java comment grammar works as advertised. Eric "J. David Beutel" wrote: Java defines two separate grammars: lexical and syntactic. The comments are in the lexical grammar, along with Unicode escapes and whitespace. It's applied separately, before the syntactic grammar. http://java.sun.com/docs/books/jls/third_edition/html/grammars.html#151185 I guess this follows the tradition of Lex and Yacc, and many languages do it the same way. I would have assumed that the distinction was for performance reasons, but since performance is so much better nowadays, I guess it also makes it easier to understand. So I wouldn't worry about unifying it for SDL. Cheers, 11011011 "Jesse Weaver" wrote on Monday, January 08, 2007 08:06:27: > I guess it'll have to be. Thanks for the advice, all. > > On 1/8/07, *Daniel Leuck* > wrote: > > Hi Eric, > > > > Thank you for the feedback. Jesse - perhaps your comment above > the definition noting that multi-line comments are not covered is > sufficient. > > > > _______________________________________________ > Sdl-developers mailing list > Sdl-developers at ikayzo.org > http://ml.ikayzo.org/mailman/listinfo/sdl-developers > > > > > -- > Jesse > > ----------- > There is a theory which states that if anybody ever discovers exactly > what the Universe is for and why it is here, it will instantly > disappear and be replaced by something even more bizarre and > inexplicable. There is another theory which states that this has > already happened. - Douglas Adams, author of Hitchhiker's Guide to the > Galaxy > ------------------------------------------------------------------------ > > _______________________________________________ > Sdl-developers mailing list > Sdl-developers at ikayzo.org > http://ml.ikayzo.org/mailman/listinfo/sdl-developers > _______________________________________________ Sdl-developers mailing list Sdl-developers at ikayzo.org http://ml.ikayzo.org/mailman/listinfo/sdl-developers "Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety." -- B. Franklin -------------- next part -------------- An HTML attachment was scrubbed... URL: From pianohacker at gmail.com Mon Jan 8 14:28:55 2007 From: pianohacker at gmail.com (Jesse Weaver) Date: Mon, 8 Jan 2007 16:28:55 -0800 Subject: [Sdl-developers] ABNF Grammar In-Reply-To: <251844.35644.qm@web50409.mail.yahoo.com> References: <45A296D4.5010702@getsu.com> <251844.35644.qm@web50409.mail.yahoo.com> Message-ID: Just in case it makes a difference, I'm writing a completely handcoded recursive descent parser for the C++ version (Boost::Spirit is a lot of library overhead). Right now, the parsing functions for each part of the grammar will look at the text stream directly. Should I add a lexer-type class, and implement the multiline comments in there, or simply write in multiline comments as a fallback when nothing else can be matched? -- Jesse ----------- There is a theory which states that if anybody ever discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable. There is another theory which states that this has already happened. - Douglas Adams, author of Hitchhiker's Guide to the Galaxy -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdb at getsu.com Mon Jan 8 14:53:09 2007 From: jdb at getsu.com (J. David Beutel) Date: Mon, 08 Jan 2007 14:53:09 -1000 Subject: [Sdl-developers] ABNF Grammar In-Reply-To: References: <45A296D4.5010702@getsu.com> <251844.35644.qm@web50409.mail.yahoo.com> Message-ID: <45A2E775.2040105@getsu.com> "Jesse Weaver" wrote on Monday, January 08, 2007 14:28:55: > Just in case it makes a difference, I'm writing a completely handcoded > recursive descent parser for the C++ version (Boost::Spirit is a lot > of library overhead). Right now, the parsing functions for each part > of the grammar will look at the text stream directly. Should I add a > lexer-type class, and implement the multiline comments in there, or > simply write in multiline comments as a fallback when nothing else can > be matched? I don't know. I just noticed how SDL differs from Java on this point. In Java the newlines don't matter; they're just whitespace, except with the single-line comments. But in SDL the newlines also separate children, so they must be parsed at the syntactic level. In other words, I haven't given this enough thought, so I'll stop commenting. Cheers, 11011011 From dan at ikayzo.com Mon Jan 8 14:54:21 2007 From: dan at ikayzo.com (Daniel Leuck) Date: Mon, 8 Jan 2007 14:54:21 -1000 Subject: [Sdl-developers] ABNF Grammar In-Reply-To: Message-ID: Hi Jesse, I would just write a method to consume the multiline comments during the tokenization phase. At this point, I don't think SDL's grammar is complex enough to require use of a parser generator. I considered using JJTree as we do for BeanShell (https://javacc.dev.java.net/doc/JJTree.html), but I discovered writing it by hand wasn't too hard, and the Java and C# implementations are very fast. Dan _____ From: sdl-developers-bounces at ikayzo.org [mailto:sdl-developers-bounces at ikayzo.org] On Behalf Of Jesse Weaver Sent: Monday, January 08, 2007 2:29 PM To: sdl-developers at ikayzo.org:Information relating to SDL development Subject: Re: [Sdl-developers] ABNF Grammar Just in case it makes a difference, I'm writing a completely handcoded recursive descent parser for the C++ version (Boost::Spirit is a lot of library overhead). Right now, the parsing functions for each part of the grammar will look at the text stream directly. Should I add a lexer-type class, and implement the multiline comments in there, or simply write in multiline comments as a fallback when nothing else can be matched? -- Jesse ----------- There is a theory which states that if anybody ever discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable. There is another theory which states that this has already happened. - Douglas Adams, author of Hitchhiker's Guide to the Galaxy -------------- next part -------------- An HTML attachment was scrubbed... URL: From dan at ikayzo.com Mon Jan 8 14:55:43 2007 From: dan at ikayzo.com (Daniel Leuck) Date: Mon, 8 Jan 2007 14:55:43 -1000 Subject: [Sdl-developers] ABNF Grammar In-Reply-To: <45A2E775.2040105@getsu.com> Message-ID: > -----Original Message----- > From: sdl-developers-bounces at ikayzo.org [mailto:sdl-developers- > bounces at ikayzo.org] On Behalf Of J. David Beutel > Sent: Monday, January 08, 2007 2:53 PM > To: sdl-developers at ikayzo.org:Information relating to SDL development > Subject: Re: [Sdl-developers] ABNF Grammar > > "Jesse Weaver" wrote on Monday, January 08, 2007 > 14:28:55: > > Just in case it makes a difference, I'm writing a completely handcoded > > recursive descent parser for the C++ version (Boost::Spirit is a lot > > of library overhead). Right now, the parsing functions for each part > > of the grammar will look at the text stream directly. Should I add a > > lexer-type class, and implement the multiline comments in there, or > > simply write in multiline comments as a fallback when nothing else can > > be matched? > > I don't know. I just noticed how SDL differs from Java on this point. > In Java the newlines don't matter; they're just whitespace, except with > the single-line comments. But in SDL the newlines also separate > children, so they must be parsed at the syntactic level. That's an excellent point. Newlines in SDL are significant, just as they are in languages like Python. Dan From pat at pat.net Tue Jan 9 16:42:19 2007 From: pat at pat.net (Patrick Niemeyer) Date: Tue, 9 Jan 2007 20:42:19 -0600 Subject: [Sdl-developers] ABNF Grammar In-Reply-To: <20070109005225.4C1925800CD@crazymonkey.lunch.org> References: <20070109005225.4C1925800CD@crazymonkey.lunch.org> Message-ID: Let me know if there is anything I can look at... I can tell you what we accept in the bsh JavaCC grammar, but yes, it should be simple to skip during tokenization. Pat On Jan 8, 2007, at 6:54 PM, Daniel Leuck wrote: > Hi Jesse, > > > > I would just write a method to consume the multiline comments > during the tokenization phase. At this point, I don?t think SDL?s > grammar is complex enough to require use of a parser generator. I > considered using JJTree as we do for BeanShell (https:// > javacc.dev.java.net/doc/JJTree.html), but I discovered writing it > by hand wasn?t too hard, and the Java and C# implementations are > very fast. > > > > Dan > > > > From: sdl-developers-bounces at ikayzo.org [mailto:sdl-developers- > bounces at ikayzo.org] On Behalf Of Jesse Weaver > Sent: Monday, January 08, 2007 2:29 PM > To: sdl-developers at ikayzo.org:Information relating to SDL development > Subject: Re: [Sdl-developers] ABNF Grammar > > > > Just in case it makes a difference, I'm writing a completely > handcoded recursive descent parser for the C++ version > (Boost::Spirit is a lot of library overhead). Right now, the > parsing functions for each part of the grammar will look at the > text stream directly. Should I add a lexer-type class, and > implement the multiline comments in there, or simply write in > multiline comments as a fallback when nothing else can be matched? > > -- > Jesse > > ----------- > There is a theory which states that if anybody ever discovers > exactly what the Universe is for and why it is here, it will > instantly disappear and be replaced by something even more bizarre > and inexplicable. There is another theory which states that this > has already happened. - Douglas Adams, author of Hitchhiker's Guide > to the Galaxy > > Pat -------------- next part -------------- An HTML attachment was scrubbed... URL: