Pardoux and Peng [74] introduced the general nonlinear backward stochastic differential equations(BSDEs for short)in 1990. According to these authors, the solutions of BSDEs consist of a pair of adapted processes (Y, Z) and satisfy-dY(t)=f(t,Y(t), Z(t))dt-Z(t)·dW(t), Y(T)=ζ, where W(t) is a standard Brownian motion,f is the generator function,ζ is an FT-measurable random variable. Since its wide applications in different mathematical fields, such as mathematical finance, stochastic control and stochastic differential games, BSDEs has become very important in stochastic analysis. Specially, BSDEs gives us a probabilistic formula to solutions of partial differential equations(PDEs for short), this can be viewed as nonlinear version of Feynman-Kac formula. Based on the great work of Pardoux and Peng [74], other forms of BSDEs have been studied by many authors. In 1997, El Karoui, Kapoudjian and Pardoux et al. [32] generalized BSDEs to BSDEs with reflection (reflected BSDEs for short), that is, to a setting with another continuous, increasing process in the equation; the function of the increasing process is to keep the solution process above a certain prescribed lower-boundary process and to do this in a minimal fashion. Later, in Cvitanic and Karatzas [26], they obtained existence and uniqueness for BSDEs with two reflecting barriers, that is, in addition to agreeing with the terminal condition at the terminal time t= T, the solution has to stay between two prescribed upper-boundary and lower-boundary processes, almost surely.Recently, motivated by capital distribution curve in stochastic portfolio theory, rank-based stochastic differential equations(rank-based SDEs for short) has received much attention. This is a new kind of SDEs, in which, the drift and diffusion coefficients of each component (particle) are decided by its rank in the vector of all components of the solution, thus, piecewise constant. The ranked particles derived from rank-based SDEs is a kind of semimartingale reflected Brownian motion, whose equations are determined in Banner and Ghomrasni [3].In the first four chapters, we will study rank-based forward backward stochas-tic differential equations. Specially, we will study decoupled rank-based forward backward stochastic differential equations and its relation with partial differential equations with Neumann boundary conditions in Chapter 1; in Chapter 2, we will study rank-based reflected backward stochastic differential equations and its relation with obstacle problems with Neumann boundary condition; in Chapter 3, we will study European option pricing and America option pricing; in Chapter 4, we will study stochastic differential games. Unlike the partial differential equations studied in Crandall, Ishii and Lions [25], whose domain has twice continuously differentiable boundary, the domain of partial differential equations in the first four chapters only has Lipschitz continuous boundary.Based on Lebesgue measure and Lebesgue integral, Kolmogorov first defined probability theory in 1933. Because its wide applications in other fields, probability theory becomes a very important part of mathematics. This classical probability theory is based on linear probability or linear expectation. However, many uncertain phenomenon can not be described by linear probability or linear expectation, for example, the Allias and Ellesberg Paradox. Therefore, nonlinear probability is used by many scholars. Choquet integral was introduced in 1953 by Choquet. Based on BSDEs, Peng introduced g-expectation in 1997. After that, Peng introduced nonlinear expectation theory in 2006.In Chapter 5, we will study large deviation principle under sublinear expecta-tion. First, we prove that large deviation principle for negatively dependent random variables under sublinear expectation, then we use it to obtain the upper bound of moderate deviation principle. In Chapter 6, we study ergodic theorem under capacity.The thesis consists of 6 chapters, whose main results are summed up as follows:(I) In Chapter 1, we study rank-based forward backward stochastic differential equations.In §1.1, we introduce the following rank-based stochastic differential equations: Then we prove the following properties for named particles and ranked particles:Theorem 1.3 For every T> t≥0 and p≥1, there exist two constants C1 and C2 depending on (p, T,{bj}) and (p, n,{bj}),respectively, such that for every x, x’∈Γn and t,t’∈[0, T], we have andTheorem 1.4 For every T> t≥ 0 and p≥1, there exist two constants C1 and C2 depending on (p, T, n,{bj}) and (p, n,{bj}). respectively, such that for every x, x’∈ Γn and t,t’∈ [0, T], we have andIn §1.2, we study the following rank-based forward backward stochastic differ-ential equations: After giving some necessary conditions, we prove the existence and uniqueness of the solution.In §1.3,we study the viscosity solution of the following PDEs:This is a new kind of PDEs, where, the solution satisfied Cauchy condition at the terminal time and Neumann boundary condition on Lipschitz continuous boundary. The main results in this subsection is as follows:Theorem 1.8 Suppose (H1.1) and (H1.2) hold. The function u defined in (1.2.12) is a viscosity solution of (1.3.1).Theorem 1.9 Suppose (H1.1), (H1.2) and (H1.3) hold. There exists at most one viscosity solution u of (1.3.1) such that for some A> 0.In §1.4, we will study BSDEs with Brownian particles with asymmetric collision-s. First, we prove the following properties for Brownian particles with asymmetric collisions:Theorem 1.10 For every T> t≥0 andp> 1, there exists a constant C depending on (L,p,T,n,{bi},{σi) such that for every x,x’ ∈ Γn andt,t’ ∈[0,T], we have andBased on these estimations, we study the connection between BSDEs with Brownian particles with asymmetric collisions and PDEs. The main conclusion in this subsection is as follows:Theorem 1.12 Suppose PDEs (1.4.13) has a solution which belongs to C1,2(0,T] x Γn;R) and there exist some c,p> 0 such that, Then the solution is unique and (1.4.12) holds.Theorem 1.13 Suppose (H1.4), (H1.5) and (H1.6) hold, u(t,x) defined by (1.4.12) is the unique viscosity solution of (1.4.13) such that it satisfies (1.3.2).(II) In Chapter 2, we will study rank-based reflected backward s-tochastic differential equations.In §2.1, our main result is the following properties when the diffusions of rank-based SDEs is not constant:Theorem 2.1 Suppose the sequence (0, σ12,...,σn2,0) is concave, then there exists a unique strong solution of the system (2.1.1) defined for allt> 0. Moreover, for all T> t≥0 and p> 1, there exists a constant depending on (p,T,{δj},{σj}) such that for any x, x’ ∈Γn and t,t’ ∈[0, T], we have andTheorem 2.2 For allT> t≥0 and p> 1, there exists a constant depending on (p, T, n,{δj},{σj}) such that for any x, x’ ∈Γn and t,t’ ∈[0, T], we have andIn §2.2, we introduce rank-based reflected backward stochastic differential e-quations and obtain the existence and uniqueness of the solution: Theorem 2.4 Suppose (H1.1), (H1.2) and (H2.1) hold, there exists a unique triple (Yt,x,Zt,x,Kt,x) of progressively measurable processes such that (i){Yt,x,Zt,x,Kt,x) satisfy (2.2.1); (ⅱ) E∫tT(|Yt,x(s)|2+|Zt,x(S)|2)ds<∞ (ⅲ) Yt,x(s)≥h(s,Xt,x(s)),t≤s≤T; (ⅳ){Kt,x(s)} is a continuous increasing process andIn §2.3, we obtain the existence and uniqueness of viscosity solution of obstacle problems:Theorem 2.7 Suppose (H1.1), (H1.2) and (H2.1) hold, u(t,x) defined by (2.2.5) is a viscosity solution of obstacle problems (2.3.1).Theorem 2.8 Suppose (H1.1), (H1.2), (H1.3) and (H2.1) hold..then, there exists at most one viscosity solution u of (2.3.1) such that it satisfies (1.3.2).In §2.4, we study reflected BSDEs with Brownian particles with asymmetric collisions and obtain a similar result as Theorem 2.3.3 and Theorem 2.4.4(see The-orem 2.10).(Ⅲ) In Chapter 3,we study option pricing.In §3.1, we study the European option pricing problem. The main conclusion is as follows:suppose the prices P0t,p(s),{Pit,p(s)}i=1n of these financial instruments evolve according to the following equations: Then, the value of the contingent claim ζ=g{P0t,p(T), Pt,p>p(T)) at time s is where, u(t,p) is the unique viscosity solution of the following parabolic PDEs: where,In §3.2, we compare the European option pricing problems in two markets. In a market with N+1 stocks whose processes are rank-based, if at the initial time 0 the price of a stock is sufficiently small and we only consider the first N stocks in option pricing, then it is almost the same as there are only N stocks in the market, i.e., the N+1 stock has vary small influence.In §3.3, we study American option pricing problem. The American option price has a minimal square integrable superhedging strategy and Yt,p(s) is its price pro-cess,where,{Yt,p,π) is the unique solution to the following reflected BSDEs:(III) In Chapter 4, we study zero-sum two-player stochastic differen-tial games, in which the state equations are competing Brownian particles and the cost functional is defined by generalized backward stochastic d-ifferential equations.In §4.1, we introduce generalized BSDEs and obtain the comparison theorem. We introduce the stochastic differential games and obtain dynamic programming principle in §4.2, that is Theorem 4.5 Suppose (H4-3), H(4.4),(H4.5) and (H4.6) hold, the lower value function W(t,x) satisfied the following DPP:for any 0<t<t+δ,x∈Γn, Finally, in §4.3 we study the viscosity solution of related Isaacs equations.(IV)In Chapter 5,we study large deviation principle and moder-ate deviation principle for negatively dependent random variables under sublinear expectation.In §5.1,we introduce the preliminaries on sublinear expectation.Then we give the definition of negatively dependent random variables and obtain some of their properties.In §5.2, we obtain large deviation principle for negatively dependent random variables, that is,Theorem 5.1 Let a random sequence{Xi}i=1 be negatively dependent under E[·] with E[edδ|Xi|]<+∞ for all 5> 0 and ≥ 1 and denote Sn=1/n ∑i=i Xi.Suppose (H5.1) hold.(1) For any closed set F,(2) For any open set G,In §5.3, based on the large deviation principle we proved in §5.2, we get the upper bound of moderate deviation principle: Theorem 5.2 Let a random variable sequence{Xi}i=1∞ be negatively dependent under E[·] with E[X1]= E[-X1)=0 and E[|Xi|2+δ]<+∞, E[et|xi]<+oo for some δ∈e (0,1),all i≤1 and all t∈R. Suppose moreover, if for all t ∈R and n≥1 Then, for any closed set F(?)R, where(V) In Chapter 6,we study ergodic theorem under capacity.In §6.1, we introduce the preliminaries of capacity and define measure-preserving transformation under capacity. In §6.2, we obtain the ergodic theorem under convex continuous capacity:Theorem 6.1 Let (Ω,F) be a measurable space, v ∈C(Ω) and satisfy v(A)≤ 1-v(Ac)≤ip(v(A)), where φ∈Φ. T is a measure-preserving transform on Ω, X is a random variable with E[||]<+∞. Let Then, (1) if X≥0, (2)If X≤0, (3) For any X,... |