Bug 1095

Summary: Establish self hosted forum/mailinglist on jogamp.org
Product: [JogAmp] General Reporter: Sven Gothel <sgothel>
Component: infrastructureAssignee: Sven Gothel <sgothel>
Status: CONFIRMED ---    
Severity: enhancement CC: derijcke.erik, elect86, gouessej, harvey.harrison, org.jogamp, wwalker3, xerxes, xerxes
Priority: P3    
Version: tbd   
Hardware: All   
OS: all   
Type: FEATURE SCM Refs:
Workaround: ---

Description Sven Gothel 2014-10-16 11:14:19 CEST
Mark Raynsford via
  <http://forum.jogamp.org/Forum-improvement-tp4033326p4033363.html>

<<<<
While I'm currently indifferent as to whether or not the project needs a new forum... An attempt was actually made last year due to Nabble falling over under heavy load.

Basically, the requirements of anything we host ourselves are:

* Security - The project's source code lives on that server. We have mirrors in lots of places, but any compromise would likely still be bad news.
* Support for posting over mail. Sven uses the forum as a mailing list, as he said.
* Importing data into the new system in a manner that preserves URIs. Put anything into a search engine about JOGL and you'll get hundreds of results linking to specific forum posts. That's something we don't want to mess with!

phpbb has an attrocious security history (as do the majority of PHP projects). They claim to have improved things, but I'm skeptical that systemic problems like that can ever be fixed. They claim to be able to import data from Nabble, but I don't see any claim that URIs will be preserved.

We tried punbb.org (http://punbb.org), because although it was still PHP, it was a lot smaller and had a better security history than any of the others. It was a bit too minimal though: It didn't have any support for mail or importing data from Nabble.

We then tried SMF (http://simplemachines.org). Their security history at the time was "better than phpbb", but I don't know any more than that. They did have a mailing list plugin to allow posting via mail, and it gave every appearance of working. When it came to Nabble, however...

http://jogamp.org/log/irc/jogamp_20130701050539.html#l80

So it kind of stalled there.

If there was an option that satisfied the first two requirements, I'd not personally be opposed to putting the Nabble forum in a read-only state and starting a new forum on a different subdomain (if it's even possible to do this with Nabble). This would preserve the old links but direct all new posts to the new forum.

Like I said, though, I'm indifferent. If people are picking software projects based on the software used to drive the project forum, then we probably don't need them!

>>>>>
Comment 1 Sven Gothel 2014-10-16 11:20:55 CEST
'Zuzub' via <http://jogamp.org/log/irc/jogamp_20141016050623.html#l9>

<<<<
[1] If you want to migrate, isnt it better to put the old forum 
in read-only mode so people can do searches and just start with a clean slate.
No migration pains.

[2] If people want to continue a thread in the new forum, 
just start a thread with a link to the old post.

[3] After 1 or 2 years the old forum info will be pretty obsolete 
and you can move it offline
>>>

While I don't agree w/ the obsolesce state [3],
the protocol [2] for thread continuation seems feasible.
Comment 2 Julien Gouesse 2014-10-16 14:55:20 CEST
There are already tons of links to some posts and forum sections. I would prefer that we use the same URL for the forum (forum.jogamp.org) instead of 2 forums with different URLs.

I have found no converter for Nabble here:
http://download.simplemachines.org/?converters
Comment 3 Sven Gothel 2014-10-20 13:34:51 CEST
Discourse:

<http://forum.jogamp.org/Forum-improvement-tp4033326p4033414.html>

About <http://www.discourse.org/about/>

and the FAQ <http://www.discourse.org/faq/>

Source <https://github.com/discourse/discourse>

- Uses Ruby
- Claims Mailinglist support
Comment 4 Sven Gothel 2014-10-20 13:36:44 CEST
(In reply to comment #3)
> Discourse:
> 
> <http://forum.jogamp.org/Forum-improvement-tp4033326p4033414.html>
> 
> About <http://www.discourse.org/about/>
> 
> and the FAQ <http://www.discourse.org/faq/>
> 
> Source <https://github.com/discourse/discourse>
> 
> - Uses Ruby

"Discourse is a JavaScript application that runs in your web browser, using the Ember.js JavaScript framework.

The server side of Discourse is written in Ruby on Rails with a Postgres database, and Redis server cache."

> - Claims Mailinglist support
Comment 6 Erik De Rijcke 2014-10-20 14:19:26 CEST
I'd be very careful with trying to migrate all content from one solution to another. In my experience something is *always* missing, corrupt, unusable.

Instead it might perhaps be more interesting to simply keep the old system running in a 'read only' mode for history or reference, and instead start with a clean slate.

Just my 2 cents.
Comment 7 Giuseppe Barbieri 2014-10-20 14:41:36 CEST
(In reply to comment #6)
> I'd be very careful with trying to migrate all content from one solution to
> another. In my experience something is *always* missing, corrupt, unusable.
> 
> Instead it might perhaps be more interesting to simply keep the old system
> running in a 'read only' mode for history or reference, and instead start
> with a clean slate.
> 
> Just my 2 cents.

I have also the same feeling (but not experience).

Anyway it would be also possible to do both, no? So if you are not satisfied with the ported version, you have always the possibility to go to the old read-only forum and check by yourself the original post/s

No?
Comment 8 Giuseppe Barbieri 2014-10-20 14:50:10 CEST
Regarding security, I suggest you to take a look to this

https://github.com/discourse/discourse/blob/master/docs/SECURITY.md

Anyway, I installed it on my VPS, at http://178.62.219.17/

I still need to figure it out the dns stuff, I created a .tk example just to test it, I need to finish to configure yet a couple of things and finally it should be ready.

If you want to register, some use the ip.

I will also test the mailing plugin.
Comment 9 Giuseppe Barbieri 2014-10-21 23:48:54 CEST
Hi dears ^^

It took me a while to solve my dns problems, I have to do things 3/4 times before I finally could have all working like a charm..

Anyway, take a look

http://jogamp.tk/t/keyreleased-problem/14/3

:)
Comment 10 Giuseppe Barbieri 2014-10-22 09:34:08 CEST
However, for the ones who didnt see my forum, I confirm mail posting is working.
Comment 11 Giuseppe Barbieri 2014-10-29 15:15:29 CET
I am kind of following the whiteDragon progresses with his dk2 on his mac

In these days I ask him to post on the discourse to test it for a "real case scenario"

I am pretty satisfied, it looks nice so far, pretty usable and clean

http://jogamp.tk/t/jogl-jovr-and-oculus-rift-dk2-modelview-matrix-problems/23/4
Comment 12 Giuseppe Barbieri 2014-11-04 10:45:19 CET
I am gonna put it down for the moment, if you want to test it again, let me know
Comment 13 Sven Gothel 2015-03-10 10:28:41 CET
*** Bug 762 has been marked as a duplicate of this bug. ***
Comment 14 Sven Gothel 2015-03-10 10:31:03 CET
(In reply to comment #7)
> (In reply to comment #6)
> > I'd be very careful with trying to migrate all content from one solution to
> > another. In my experience something is *always* missing, corrupt, unusable.
> > 
> > Instead it might perhaps be more interesting to simply keep the old system
> > running in a 'read only' mode for history or reference, and instead start
> > with a clean slate.
> > 
> > Just my 2 cents.
> 
> I have also the same feeling (but not experience).
> 
> Anyway it would be also possible to do both, no? So if you are not satisfied
> with the ported version, you have always the possibility to go to the old
> read-only forum and check by yourself the original post/s
> 
> No?


Yes, doing both would be desired.
Comment 15 Sven Gothel 2015-03-10 10:36:08 CET
Since we picked up this issue in forum again,
let me ask again about the following requirements:

Hard Requirements:
 - Free Software License

 - Mailinglist feature
    - Receive / Send messages via emails

 - Secure Installation Possible,
   e.g. chroot, DB separation .. dunno

 - Installation on jogamp.org

 
Optional Requirements:
 - Preserve nabble content
 - Mailinglist works as a mailinglist (mailman, ..)


Patience .. we will do this task, 
but is has surely not the highest priority in this project.

Thank you!
Comment 16 Sven Gothel 2015-03-10 10:39:17 CET
(In reply to comment #15)
> Since we picked up this issue in forum again,
> let me ask again about the following requirements:
> 
> Hard Requirements:
>  - Free Software License
> 
>  - Mailinglist feature
>     - Receive / Send messages via emails
> 
>  - Secure Installation Possible,
>    e.g. chroot, DB separation .. dunno
> 
>  - Installation on jogamp.org

   - Shall work _without_ scripting languages,
     i.e. javascript.
     (It is OK if javascript enables more stuff,
      but shall not be a requirement for the client)

So if "Discourse" cannot work properly w/o javascript,
IMHO this is a NO-GO then.
Comment 17 Julien Gouesse 2015-03-10 11:17:10 CET
> So if "Discourse" cannot work properly w/o javascript,
> IMHO this is a NO-GO then.

"Discourse is a JavaScript application":
http://www.discourse.org/faq/#browser

Please can you elaborate why Javascript is a problem for you? Personally, I use it only when it is absolutely necessary so that the end users who disable it can see most of my stuff. I don't know whether the modern crawling bots really look at the Javascript code.
Comment 18 Giuseppe Barbieri 2015-03-11 12:44:31 CET
I also don't have any particular problem against javascript
Comment 19 Mark Raynsford 2015-03-11 18:21:50 CET
I object to Javascript on philosophical grounds: It's an awful, awful language. It has semantics that can be roughly summarized as "yeah, do whatever you want". It's unsafe and an endless source of security and compatibility problems. If that wasn't bad enough, the vast majority of the time its use is completely unnecessary. It seems like the justification for it is usually "my server can't handle forking a thousand php processes at a time so lets do all the work on the client" without any thought given as to how the server side might be built in a manner that doesn't involve forking a thousand php processes.

However, I also don't think it's going to be possible to find any "web" software that isn't utterly infested with it these days (hence why I've more or less given up on "the web").
Comment 20 Sven Gothel 2015-03-13 07:48:55 CET
(In reply to comment #17)
> > So if "Discourse" cannot work properly w/o javascript,
> > IMHO this is a NO-GO then.
> 
> "Discourse is a JavaScript application":
> http://www.discourse.org/faq/#browser
> 
> Please can you elaborate why Javascript is a problem for you? Personally, I
> use it only when it is absolutely necessary so that the end users who
> disable it can see most of my stuff. I don't know whether the modern
> crawling bots really look at the Javascript code.

Looking at jmonkeyengine's forum
   <http://hub.jmonkeyengine.org/>

I do see _all_ reasons of _not_ using _not_ only 
a javascript requiring forum, but one that fetches it
javascript code from many different server!
(I.e. its code-locations are from multiple server).

I realized that while checking w/ the forum
and I always have NoScript installed, 
manually granting access to one server.
But NO - this is not enough here ..

Horrible experience.

And .. it's non javascript view is also unacceptable.

I will go back in this discussion and 
suggest the solution Mark was working on.
Comment 21 Sven Gothel 2015-03-13 08:00:07 CET
(In reply to comment #19)
> I object to Javascript on philosophical grounds: It's an awful, awful
> language. It has semantics that can be roughly summarized as "yeah, do
> whatever you want". It's unsafe and an endless source of security and
> compatibility problems. If that wasn't bad enough, the vast majority of the
> time its use is completely unnecessary. It seems like the justification for
> it is usually "my server can't handle forking a thousand php processes at a
> time so lets do all the work on the client" without any thought given as to
> how the server side might be built in a manner that doesn't involve forking
> a thousand php processes.

I agree here alot. When I was porting the JPEG decoder from javascript
to java - it was a nightmare, backtracking variable's types,
since they just don't declare them properly!

> 
> However, I also don't think it's going to be possible to find any "web"
> software that isn't utterly infested with it these days (hence why I've more
> or less given up on "the web").

See my other comment 20.
I.e. our forum should be self-hosted, meaning, 
whatever it is - it should all come from jogamp.org, 
nothing else should be required.
Hence I agree here as well, websites these days seem
to depend on too many other services and they will fail,
once those become unavailable.

Simply assume, jogamp.org shall work in your LAN
w/o internet connection!
Comment 22 Giuseppe Barbieri 2015-03-23 11:39:43 CET
I encourage you to partecipate 

https://meta.discourse.org/t/is-it-a-javascript-based-software-forum-that-bad/26651

They point out like modern websites such as Gmail, Twitter, Facebook requires Javascript

And given it is an open source software, all assets can be served from your own host
Comment 23 Julien Gouesse 2015-03-23 11:47:43 CET
(In reply to comment #22)
> I encourage you to partecipate 
> 
> https://meta.discourse.org/t/is-it-a-javascript-based-software-forum-that-
> bad/26651
> 
I'll give it a look.

> They point out like modern websites such as Gmail, Twitter, Facebook
> requires Javascript
> 
In my humble opinion, there is a difference between "modern" and "good". Moreover, Facebook and Twitter aren't good examples of services as they are highly centralized and easy to break. Twitter is often unable to handle all its tweets and Facebook can't resist to moderately big DDOS attacks. I'm not a big fan of Javascript but our opinion should depend on how it is used too (abusing of Javascript is bad for search engines indexing) and I keep in mind that this language is a "must", something unavoidable on the Web (on the client side).

> And given it is an open source software, all assets can be served from your
> own host

Good point. In your humble opinion, does it require lots of modifications to host the libraries on our server(s)?
Comment 24 Sven Gothel 2015-04-10 11:40:47 CEST
(In reply to comment #22)
> I encourage you to partecipate 
> 
> https://meta.discourse.org/t/is-it-a-javascript-based-software-forum-that-
> bad/26651
> 
> They point out like modern websites such as Gmail, Twitter, Facebook
> requires Javascript

Invalid, since we define our own requirements.
Other's choice of a platform is theirs to make.

I have read through the above discussion.

It boils down to:

- Standard HTML/CSS based [thin] Client
  _and_ EMail-List Clients

  Our client requirements shall be as simple
  as possible, i.e. only based on 
    - HTML/CSS standards
    - EMail

  Complicated client side processing,
  maybe even with downloading tools/libs
  from other hosts shall be forbidden.

  This thin client requirement will ensure
  and allow all users to participate,
  regardless of device and browser!


- Most Secure Server Side Tech
  - Less tools/libs is better

  - All tools/libs must be self-hosted,
    no 'online' fetching at all

  - Complications make things more error prone
    and hence insecure

> 
> And given it is an open source software, all assets can be served from your
> own host

Yes, that is another requirement.
Good that this might be satisfied!
Comment 25 Giuseppe Barbieri 2015-08-21 17:18:03 CEST
I would like to add that, in my opinion, keeping this forum is way worst than having a modern one with javascript. 
Speaking about device compatibility, for example, Discourse is much more suitable for modern devices (smartphones) than Nabble.

I agree that there is no point on having a fat client, that javascript is often abused and so on, but I see load work is high and I have the feeling that if we don't do something this community is gonna die sooner or later.

Forums are one of the pillars of a community and since I am a nub I can guarantee you many others nubs may see our forum as "2000 crap". I sound rude, but I want to be honest, make myself clear and presenting the things from my point of view.

My 2 cents.
Comment 26 Julien Gouesse 2015-08-24 17:34:12 CEST
I confirm that Discourse goes on working when Javascript is disabled, it's just less pretty but it's acceptable. I have just set javascript.enabled to false in "about.config" in Mozilla Firefox 40 and I have gone on the official JMonkeyEngine forum :
"Powered by Discourse, best viewed with JavaScript enabled"

In my humble opinion, maybe we should ensure that we won't lose anything important. There is nothing strongly stopping us from moving now.
Comment 27 Julien Gouesse 2015-08-26 13:12:24 CEST
It's possible to create some URL rewriting rules for Apache so that the old posts using Nabble's syntax can still be found by the search engines when switching to Discourse:
http://jogamp.org/log/irc/jogamp_20150824050531.html#l447

Then, we could keep all threads unchanged (no read-only mode) in a single forum instead of using 2 forums.
Comment 28 Julien Gouesse 2015-08-27 10:18:38 CEST
There are several means of creating permanent redirections with Apache:
- Redirect 301 /oldlocation http://www.mydomain.com/newlocation
- Redirect permanent /oldlocation http://www.mydomain.com/newlocation
- RedirectMatch ^/shhoehpeo/(.*)$ http://forum.jogamp.org/$1
- AllowOverride + RewriteEngine on + RewriteRule

https://httpd.apache.org/docs/2.2/mod/mod_rewrite.html#rewriterule
Comment 29 Giuseppe Barbieri 2015-09-25 08:29:16 CEST
http://blog.discourse.org/2015/09/discourse-1-4-released/
Comment 30 Giuseppe Barbieri 2016-02-01 11:34:46 CET
I would like to bring up the urgency of this aspect.

It really turns me down the fact that:

- there is no immediate way to see if a thread has new replies

- in a multi-page thread I can't go immediately to the last page

- no code formatting

- general aspect and layout
Comment 31 Julien Gouesse 2017-01-19 10:29:40 CET
Hi

I know that it's not the most important aspect now. I agree with Sven about "Other's choice of a platform is theirs to make." and it's true for Github too (which is occasionally blocked in some countries). We have our own requirements. 

Personally, I'd suggest XWiki, it's a second generation Wiki software, it's written in Java, it's open source, it can be self hosted, it goes on working with Javascript disabled, it's very capable, it can be used both for the wiki and for the forum (even as a file manager, a blog, a task management system, a calendar, ...). Using a Java based solution would avoid increasing the surface of attack and we obviously can fix it and adapt it to our needs but I assume that we already need Python and PHP for other stuffs :( (Git web viewer?).

If something goes wrong with a Ruby based solution, I'll be unable to help you.

Note that there is no emergency as the crashes of Nabble mentioned by Xerxes no longer occur despite the high number of users.