ADS Local Server and simultaneous requests

[ricardo arraes]

Hey everybody!

As some of you may know and some of you may not know... I had some troubles with ADS Local Server and simultaneous requests which were opening and manipulating tables...

this is the topic I created to ask for some help:

viewtopic.php?f=5&t=112

But I decided to create another topic, to give you some more precise information about the real problem.

When multiple users were sendind requests to my webservice, some of them were opening tables (sometimes different tables, sometimes the same table) at the exact same time, and for some unknown reason it was crashing apache with an ADS error.

Ok... we are talking about ADS Local Server, so can assume that there are some limitations about concurrency and connections managements. So what I did to solve this problem was creating a simple flag control via MySQL, like that:

Código: Seleccionar todo

lOpen:=.T.
oBDFlag:= TMySQl():New("MARIA","localhost","uset","psw","schema",3306)

	caliasflag:= "ALIAS01"
	nTime:=Seconds()			

	DO WHILE lOpen .AND. (Seconds()-nTime) < 30 

		oQry:=NIL

		oQry:=oBDFlag:Query("SELECT data, cflag, time FROM flag WHERE data='PARAM01' ",{Trans(Dtos(Date()),"@R 9999-99-99")})

		IF oQry = NIL .OR. oQry:Count()<=0  			

			IF (nT:=oBDFlag:Exec({"INSERT INTO flag (data, cflag, time) VALUES('PARAM01','PARAM02','PARAM03')"},;
						{{Trans(Dtos(Date()),"@R 9999-99-99"),caliasflag,Time()}})) < 0			
								
				lOpen:=.T.						
			ELSE 				

				WaitPeriod(100)

				DO WHILE WaitPeriod()
					
				ENDDO

				oQryV:=NIL
				oQryV:=oBDFlag:Query("SELECT cflag FROM flag WHERE data='PARAM01'",{Trans(Dtos(Date()),"@R 9999-99-99")})
				oQryV:First()

				IF oQryV:FieldGet("cflag")=caliasflag					
					lOpen:=.F.				
				ELSE 
					lOpen:=.T.														
				ENDIF
				
			ENDIF								

		ELSE
			oQry:First()
			
			IF (Seconds()-Secs(oQry:FieldGet("time")))>=20
				
				
				oBDFlag:Exec({"DELETE FROM flag WHERE data='PARAM01'"},{{Trans(Dtos(Date()),"@R 9999-99-99")}})	
				
				IF (oBDFlag:Exec({"INSERT INTO flag (data, cflag, time) VALUES('PARAM01','PARAM02','PARAM03')"},;
					{{Trans(Dtos(Date()),"@R 9999-99-99"),caliasflag,Time()}})) < 0			
					
					lOpen:=.T.
				ELSE 

					WaitPeriod(100)

					DO WHILE WaitPeriod()
				
					ENDDO

					oQryV:=NIL
					oQryV:=oBDFlag:Query("SELECT cflag FROM flag WHERE data='PARAM01'",{Trans(Dtos(Date()),"@R 9999-99-99")})
					oQryV:First()

					IF oQryV:FieldGet("cflag")=caliasflag
						lOpen:=.F.	
											
					ELSE 
						lOpen:=.T.

					ENDIF
					
				ENDIF	

			ELSE 				
				lOpen:=.T.
				
			ENDIF
		ENDIF	

		
		WaitPeriod(200)
	
		DO WHILE WaitPeriod()
				
		ENDDO					
		
	ENDDO
	
	IF lOpen
		
		hResponse[ 'success' ] := .T.
		hResponse['cod'] := "TIMEOUT"
	
		oController:oResponse:SendJson( hResponse )	
		RETURN NIL
	
	ENDIF
	
	

Basically, when the webservice receives a request it will set a flag value so any other request must wait until the first one finishes it's processing.

Well, not a fancy solution, but it worked... now I'm not getting apache crashes with ADS Local Server but I'm getting some "errors" that I will discuss in another topic so we can separate the situations!

I haven't tested with the ADS Remote Server, I believe that there won't be any problem because ADS Remote Server has a better management than ADS Local Server, but I will test it soon.

Hope it helps you guys!

[charly]

Ricardo,

Thanks for you explanationa about your problem. I recommend that you look at this tip, which is very interesting, because it talks about the use of aliases in the threads environment in which our friend Rafa explains to us how to manipulate them correctly so that they "don't step on each other." We have to think that we are in a thread environment in modHarbour

http://forums.fivetechsupport.com/viewt ... 37#p164261

Observe the use of
hb_dbDetach( cAlias ) // Libero el alias
and
hb_dbRequest( cAlias, , , .T.) // Restaura el alias

[ricardo arraes]

Thank you Charly!

I just read the topic you suggested...

ok, I got it. hb_detach is releasing the alias and hb_dbrequest is requesting the alias that already exists...

but, I'm not sharing aliases, I'm creating them dynamically (and "randomly") as we discussed previously. so... are these methods still worth it in this situation?

Rafa also posted a doc explaining a little bit more about harbour multithread...

https://docs.google.com/document/d/1xxR ... 0MVg/edit#

and it says that there's no problem creating other workareas inside other threads, they would be processed separately and independently (as long as the table is not locked).

But let try this! I'll update this topic with some more information about this situation

[Cristobal]

Ricardo, tal y como yo lo entiendo, a ver si consigo explicarme, el problema no es controlar el modo en el que usas los alias, ya que tú no los estás lanzando en distintos hilos, y, además, considero que el uso de alias dinámicos es lo apropiado siempre, sino que el problema lo encontramos en algo que quizás no podemos controlar, y es que necesitamos que estas peticiones se realicen en el orden que las enviamos, porque habitualmente, unas peticiones dependen del resultado obtenido de la anterior, y, es el modo en el que estos request que realizamos son procesados por el servidor, y no podemos controlar la asignación de prioridades que establece el servidor ni los tiempos asignados a las peticiones que recibe.
Por otra parte, podemos usar lo que estáis comentando si queremos compartir un alias entre distintos hilos, que aquí tampoco es necesario, ya que tal y como entiendo que realizas las peticiones, abres y cierras la base de datos cada vez, asignándole un alias dinámico.
Podemos, como comentó Charly en otro post, establecer unos tiempos de espera entre peticiones para "asegurarnos" que dichas peticiones se realizan en el orden que deseamos, pero sigue siendo un problema establecer de antemano cuánto tiempo va a necesitar cada petición, con lo que nos podríamos encontrar en la misma situación si el tiempo asignado no ha sido suficiente.
Una forma que considero posible para evitar esto es realizar las peticiones via AJAX, configurando la petición como "NO ASINCRONA", por lo que ya estamos asegurando que no se enviará una petición nueva hasta que la anterior haya terminado.
Saludos

[charly]

Hi

Una forma que considero posible para evitar esto es realizar las peticiones via AJAX, configurando la petición como "NO ASINCRONA", por lo que ya estamos asegurando que no se enviará una petición nueva hasta que la anterior haya terminado.

Se nos escapa algo.

El tema ajax sycronous resolveria que tu no provocaras mas de una peticion simultanea, pero no si dos usuarios diferentes realizan una peticion justo en el mismo momento.

Se lo dije a Ricardo, la unica manera es crear un ejemplo para probar e investigar. Si alguien lo crea de tal manera que podamos reventar el mod, me apunto a investigar...

[Cristobal]

Hi
Se nos escapa algo.

El tema ajax sycronous resolveria que tu no provocaras mas de una peticion simultanea, pero no si dos usuarios diferentes realizan una peticion justo en el mismo momento.

Bueno, creo que hay que diferenciar el solucionar el problema de la ejecución de procesos ( request ) de un solo usuario, que creo que era el problema que planteaba Ricardo, con el hecho de la "concurrencia" de varios usuarios. Ahí, evidentemente tendremos que echar mano de nuestros conocimientos a la hora de compartir ficheros, etc. y fundamentalmente utilizar algo como ADS que imagino que será el motivo de usarlo Ricardo

[ramirezosvaldo]

Pregunta:

Pero todo este problema es por modharbour ? es como modharbour implementa los threads ?

o bien, es propio de ADS ?

Saludos
Osvaldo Ramirez

[ricardo arraes]

Thank you Cristobal and Charly!

everything you guys just said makes completely sense, and probably setting Ajax as you described could solve the problem for one user, but in this situation I'm actually talking about many users sending requests... I haven't done any test about these Ajax settings you suggested because I'm always kinda intrigued about this specifically ADS situation, so I decided to keep studying and putting working on it.

Thank you again for you the effort you guys put in helping me with this issue.

Well, I guess I FINALLY solved everything related to this issue and I'll describe everything in a moment.

*I'll create a simple example so everybody can try this situation as I promised before...

[ricardo arraes]

Pregunta:

Pero todo este problema es por modharbour ? es como modharbour implementa los threads ?

o bien, es propio de ADS ?

Saludos
Osvaldo Ramirez

Hey Osvaldo!

well, after some days working on this issue, I can say that this is not a modharbour problem... In fact, everything related to ADS and DBF manipulation ran fine with modharbour...

The situation here is that we are working with web applications right now and probably most of the web application deal with (a lot of) concurrency between users, threads and stuff... So when I first tried to implement an web application to manipulate an ADS Local Server and it's DBF Files from an old desktop application of mine, using the same logic from the desktop application, I immediately fell into this situation and realized that my desktop application basically basically faces no concurrency... so, obviously, I started to have some problems.

web applications basically are executed in the same machine, which means that the same USER is manipulating the files, but every request is a "new process" (we can compare it to a multithread environment). it's a completely different scenario from a desktop application running on a local network where every computer (different users) is executing the application as a single instance.

And also, as I said, we are dealing with the ADS Local Server, which has way less features than the ADS Remote Server to control and manage the database and files....

So, basically, after all this work, we can say - not 100% sure, but at least 80% - that there's actually no error in modharbour/apache/ADS behaviour... it's just that this situation required some adaptation to suit the application to this new environment!

[ricardo arraes]

So how did I solve this whole situation...

As I said before, I have two different methods that are available to the user and they are asynchrunous, which means that, at any moment 2 or more users can executed these methods simultaneously...

just for the record, these methods are manipulating DBF files from a ADS Local Server.

So these first steps are identifical for both methods:

FIRST STEP

the first thing both these methods should do is control the concurrency between users, in order to prevent different users from opening the same file at the exact same time, which is crashing apache with an 6005-ACCESS_VIOLATION error...
So, basically what did was creating an simple mysql database with a table that will work as a flag, controlling the access to the method.
Let's assume that we got 3 users and each one of them is sending a request to the application at the same time... the very first request will insert a row with some information into the flag table and the other two requests will wait until the first finish it's processing and delete the same row it just created, so the second request will insert a row and the third request will wait, and it goes on...

*if anything goes wrong, I defined a "timeout" where 20 seconds is the most that a request can hang...

Código: Seleccionar todo

caliasflag:= StrTran(Time(),":")+cSalt+"01"
	nTime:=Seconds()			
	
	oBDFlag:= TMySQl():New("MARIA","localhost","root","psw","flag_db",3306)

	DO WHILE lOpen .AND. (Seconds()-nTime) < 30 

		oQry:=NIL

		oQry:=oBDFlag:Query("SELECT data, cflag, time FROM flag WHERE data='PARAM01'",{Trans(Dtos(Date()),"@R 9999-99-99")})

		IF oQry = NIL .OR. oQry:Count()<=0  			

			IF (nT:=oBDFlag:Exec({"INSERT INTO flag (data, cflag, time) VALUES('PARAM01','PARAM02','PARAM03')"},;
						{{Trans(Dtos(Date()),"@R 9999-99-99"),caliasflag,Time()}})) < 0			
								
				lOpen:=.T.						
			ELSE 				

				WaitPeriod(100)

				DO WHILE WaitPeriod()
					
				ENDDO

				oQryV:=NIL
				oQryV:=oBDFlag:Query("SELECT cflag FROM flag WHERE data='PARAM01'",{Trans(Dtos(Date()),"@R 9999-99-99")})
				oQryV:First()

				IF oQryV:FieldGet("cflag")=caliasflag					
					lOpen:=.F.				
				ELSE 
					lOpen:=.T.														
				ENDIF
				
			ENDIF								

		ELSE
			oQry:First()
			
			IF (Seconds()-Secs(oQry:FieldGet("time")))>=20
				
				
				oBDFlag:Exec({"DELETE FROM flag WHERE data='PARAM01'"},{{Trans(Dtos(Date()),"@R 9999-99-99")}})	
				
				IF (oBDFlag:Exec({"INSERT INTO flag (data, cflag, time) VALUES('PARAM01','PARAM02','PARAM03')"},;
					{{Trans(Dtos(Date()),"@R 9999-99-99"),caliasflag,Time()}})) < 0			
					
					lOpen:=.T.
				ELSE 

					WaitPeriod(100)

					DO WHILE WaitPeriod()
				
					ENDDO

					oQryV:=NIL
					oQryV:=oBDFlag:Query("SELECT cflag FROM flag WHERE data='PARAM01'",{Trans(Dtos(Date()),"@R 9999-99-99")})
					oQryV:First()

					IF oQryV:FieldGet("cflag")=caliasflag
						lOpen:=.F.	
											
					ELSE 
						lOpen:=.T.

					ENDIF
					
				ENDIF	

			ELSE 				
				lOpen:=.T.
				
			ENDIF
		ENDIF	

		
		WaitPeriod(100)
	
		DO WHILE WaitPeriod()
				
		ENDDO					
		
	ENDDO

	IF lOpen
		hResponse[ 'success' ] := .F.		

		oController:oResponse:SendJson( hResponse )	
		RETURN NIL
	
	ENDIF
	
	
	---PROCESSING STUFF-----
	
	oBDFlag:Exec({"DELETE FROM flag WHERE data='PARAM01'"},{{Trans(Dtos(Date()),"@R 9999-99-99")}})				
	

this will control the concurrency and, as I said, prevent some kind of collision...


SECOND STEP

after the concurrency is under control, I gotta set the connection to the ADS (Local Server) in order to manipulate files properly...

*The settings for ADS Remote Server are not properly configured, I haven't tested with ADS Remote server yet...

So immediately after the code from the first step I used this code right here to properly set my application to use ADS Local Server:

Código: Seleccionar todo

DO WHILE .T.

		IF AdsIsServerLoaded( Left(cDirAds,2), @nConn ) = ADS_REMOTE_SERVER
			AdsSetServerType(2)
			AdsSetFileType( 2 )
			AdsMgConnect( Left(cDirAds,2),,,@nConn )
			aTemp := AdsMgGetInstallInfo()
			cAdsSerial := IF(Len(aTemp)>=8,aTemp[8],"")
			aTemp := NIL
			nAdsConn := nConn			
			EXIT
		ELSEIF .NOT. lForcaAds
			RddRegister("ADS",1)
			RddSetDefault( "ADS" )
			AdsSetServerType(1)
			AdsSetFileType( 2 )
			
			lcon:=ADSConnect60( cDirDbf, 1, "", "",,@hAds)
			IF .NOT. lcon 
				hResponse[ 'success' ] := .F.

				oController:oResponse:SendJson( hResponse )
				RETURN NIL
			ENDIF
			
			EXIT
		ELSEIF n = 1
			n := 2			
		ELSE			
			hResponse[ 'success' ] := .F.
			hResponse['cod'] := "7"
			
			oController:oResponse:SendJson( hResponse )
			RETURN NIL
		ENDIF
	ENDDO

	caliasage:= NewAlias("PS"+hData['teste'])
	
	
	----------MORE CODE-----------------
		
	
	AdsDisconnect(hAds)
	
	

with that been said, with these two steps I'm controlling the concurrency and setting my ADS Local Server connection and now I can properly open tables and do whatever I want with them...

in the next post I'll paste an example of these two methods so you guys can take a better look!